You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Marc Perkel <ma...@perkel.com> on 2006/06/09 22:19:01 UTC
The Future of Email is SQL
After considerable experimenting and thinking things through I thought
I'd start a thread on the future of email to start planting the seeds of
where MTA development needs to go. I'm convinced that someday soon we
will all realize that MBOX and MAILDIR are obsolete technologies and
that the future is going to be SQL based storage.
First - before everyone starts screaming about speed comparisons, I'm
not going to go there. Every storage technology has it's advantages and
disadvantages but I'm just going to say that SQL based mail storage is
fast enough. The advantages of SQL has to do with power and not with
speed. Those who would choose it would do so because they want to do new
things that you can do with a database and can't do without one.
SQL has several advantages. You don't have t deal with the quirks of the
underlying file system or OS. It takes care of all the locking issues
and indexing and makes it so that multiple applications can seamlessly
access the data. With an SQL backend email can be stored from the MTA,
read from and IMAP client that accesses the same database, and the spam
filtering engine will have access to the stored email as well.
To give you some examples of what could be done .....
Suppose a spammer sends 1000 phishing spams to your users and then you
figure out that the 1000 spams already delivered is spam. With a
database you can do a query to retroactively delete spam that was
already delivered to the mailboxes. This could also be used to
retroactively delete viruses already delivered.
Spam filtering programs can lookup existing email in existing folders
and compare it with new email already deliverd to help determine more
accurately if a message is spam or not. For example, if the host server
has a reputation for 100% ham then it can deliver new email without
running it through Spam Assassin. If programs like Spamassassin can
access existing email in existing folders it can evaluate new email
using tricks no one has yet considered.
SQL databases allow for multiple masters and slaves and replication that
lets you create a cluster that never fails under any conditions. It
would be far easier to create a system that is always on and always
backed up.
An SQL backend allows you to use a wide variety of tools, programming
languages, operating systems in order for you to easily integrate more
easily than non database systems.
And - this is important - once you have a database then new things that
no one has yet thought of will be possible and new things we've never
heard of will be developed because the new power will lend to the
development of more tricks than you can do without database power.
My point here is - think outside the box. I'm going to be lobbying IMAP
server developers to include SQL backends. exim could pipe data into a
local delivery agent, or it can have features written to write directly
to the SQL backend.
Thoughts ..... ?
-- ## List details at http://www.exim.org/mailman/listinfo/exim-users ##
Exim details at http://www.exim.org/ ## Please use the Wiki with this
list - http://www.exim.org/eximwiki/
Re: The Future of Email is SQL
Posted by jdow <jd...@earthlink.net>.
From: "NM Public" <ag...@nm.deflexion.com>
> Sur 2006-06-09, Marc Perkel skribis:
>>
>> Perhaps the headers and other information that you would index
>> be kept in the database and the body of the message stored
>> somewhere else, perhaps even as files.
>
>
> It seems that this is what Zimbra does. Check out my blog post
> here:
>
> For IMAP, "SQL just sucks"
> <http://deflexion.com/2006/06/for-imap-sql-just-sucks>
>
> especially the comment from KevinH of Zimbra, which includes
> this:
>
> "It's true that as a mailstore SQL sucks. Databases are not
> designed to store large blobs of data. However in Zimbra's case
> messages aren't stored in the database. Only *meta* data is
> stored behind a SQL interface. [...]"
>
> I hope this is useful. Thanks for inspiring interesting
> discussion Marc!
> Nancy
> (sent via gmane.mail.spam.spamassassin.general)
If you only own a hammer and (think) you know how to use a hammer
all jobs seem to turn into jobs that need a hammer to solve them.
SQL is just one more tool. Don't be pathetic and try to use a hammer
as a means to remove a tiny delicate screw. You might as well try
saving email as XML objects or compiled into Objective C source
code modules. This discussion has gone on long enough it's rather
plain that SQL is not a natural fit for email storage, hasn't it?
{^_^}
Re: The Future of Email is SQL
Posted by kbaker <kb...@missionvi.com>.
Jim C. Nasby wrote:
> On Sat, Jun 10, 2006 at 01:23:35PM -0600, qqqq wrote:
>>> I would defer to the smart people to figure out the details. However I do
>>> wonder if the actual body content of the message would be best stored in a
>>> file and the SQL used to store anything and everything you would want to
>>> index. That would keep the SQL file size down if that's an issue. However,
>>> SQL databases might have to be changed to accomodate the needs to store
>>> email.
So I missed the beginning of the thread, but thought I'd point out www.dbmail.org
This is an open source IMAP server with a RDMS message store. So yes it has been
done, yes it works and yes there are a number of production installs that have
been running great for a number of years now.
It might not be as fast as Cyrus for instance, but we've been running it with
MySQL replication to second server for a fail-over for a bit now. Works great
and was *really* easy to setup.
>> I think this is what I was getting at early in the thread. I would think
>> that a 5 MB body would do better on file but I don't know enough in regards
>> to DBs to even make a call.
>
> A good rule of thumb about storing something in the database is: are you
> going to search that data? If you're going to search the text of an
> email body, that makes it a more likely candidate for storing it in the
> database (though there are ways to do this searching while storing the
> file externally).
>
> Another consideration is that storing everything in the database is
> substantially easier than splitting between a database and the
> filesystem. If you think this is a non-issue, consider how to deal with
> all the error conditions where either the database or the filesystem is
> updated, but not both.
>
> Of course, storing anything in a database is going to have more
> overheard than storing it as raw bytes on the filesystem, and there's
> not really a way around that. Different databases will impose different
> amounts of overhead.
>
> As for all the arguments about how databases won't scale, or how they're
> a single point of failure.. what exactly do you think a single mail
> server is? Answer: not scalable and a single point of failure. Of
> course there are ways to work around that, and those methods apply just
> as well to databases (though the implementation can be different). Most
> databases support at least some form of replication, and many support
> clustering. And of course you don't have to try and cram all your users
> into a single database.
>
> Having said all that; it's nearly impossible to get a general-purpose RDBMS
> to outperform an optimized storage format (if you find an example where
> it is possible, I'd wager that's only true because the original format
> wasn't very well thought-out). It's essentially a given that a given set
> of hardware will be able to handle a higher load of storing and
> retrieving emails using maildir rather than a database (unless you get
> enough messages in a directory that it starts choking the filesystem).
> But if you want to do something like search for specific emails, there's
> a much better chance that a database will outperform maildir, especially
> if you're searching the message body. And there's other potential
> applications where a database would outperform maildir as well.
>
> So, in a nutshell, if you're not going to try doing something more
> advanced than just storing and retrieving email, it's unlikely that
> you'll be happy with storing that email in a database. The further off
> that 'beaten path' you get, the more likely you are to see benefit from
> using a database.
Re: The Future of Email is SQL
Posted by Jason Haar <Ja...@trimble.co.nz>.
Jim C. Nasby wrote:
> Having said all that; it's nearly impossible to get a general-purpose RDBMS
> to outperform an optimized storage format
Indeed. I refer back to the wondrous success Microsoft Exchange has had.
It *isn't* SQL. It's a hand-crafted, JET-backend specifically written to
be optimized for email handling. Even Microsoft (so far) hasn't managed
to get their own MS-SQL product to be usable as a backend. Believe me,
they'd like nothing better from a marketing perspective...
...However, I do believe they are still working towards such an end.
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: The Future of Email is SQL
Posted by David Landgren <da...@landgren.net>.
Jim C. Nasby wrote:
> On Sat, Jun 10, 2006 at 01:23:35PM -0600, qqqq wrote:
>>> I would defer to the smart people to figure out the details. However I do
>>> wonder if the actual body content of the message would be best stored in a
>>> file and the SQL used to store anything and everything you would want to
>>> index. That would keep the SQL file size down if that's an issue. However,
>>> SQL databases might have to be changed to accomodate the needs to store
>>> email.
>> I think this is what I was getting at early in the thread. I would think
>> that a 5 MB body would do better on file but I don't know enough in regards
>> to DBs to even make a call.
>
> A good rule of thumb about storing something in the database is: are you
> going to search that data? If you're going to search the text of an
> email body, that makes it a more likely candidate for storing it in the
> database (though there are ways to do this searching while storing the
> file externally).
SQL databases suck dead dogs through a straw on full text searches. The
language specification isn't designed for it. Database vendors offer
support for it in various mutually-incompatible ways.
It's easy to precalc search indices for maildir and produce fast search
results; mairix is one such tool that I know of, there are no doubt many
other solutions available.
David
--
Much of the propaganda that passes for news in our own society is given
to immobilising and pacifying people and diverting them from the idea
that they can confront power. -- John Pilger
Re: The Future of Email is SQL
Posted by "Jim C. Nasby" <de...@decibel.org>.
On Sat, Jun 10, 2006 at 01:23:35PM -0600, qqqq wrote:
> >I would defer to the smart people to figure out the details. However I do
> >wonder if the actual body content of the message would be best stored in a
> >file and the SQL used to store anything and everything you would want to
> >index. That would keep the SQL file size down if that's an issue. However,
> >SQL databases might have to be changed to accomodate the needs to store
> >email.
>
> I think this is what I was getting at early in the thread. I would think
> that a 5 MB body would do better on file but I don't know enough in regards
> to DBs to even make a call.
A good rule of thumb about storing something in the database is: are you
going to search that data? If you're going to search the text of an
email body, that makes it a more likely candidate for storing it in the
database (though there are ways to do this searching while storing the
file externally).
Another consideration is that storing everything in the database is
substantially easier than splitting between a database and the
filesystem. If you think this is a non-issue, consider how to deal with
all the error conditions where either the database or the filesystem is
updated, but not both.
Of course, storing anything in a database is going to have more
overheard than storing it as raw bytes on the filesystem, and there's
not really a way around that. Different databases will impose different
amounts of overhead.
As for all the arguments about how databases won't scale, or how they're
a single point of failure.. what exactly do you think a single mail
server is? Answer: not scalable and a single point of failure. Of
course there are ways to work around that, and those methods apply just
as well to databases (though the implementation can be different). Most
databases support at least some form of replication, and many support
clustering. And of course you don't have to try and cram all your users
into a single database.
Having said all that; it's nearly impossible to get a general-purpose RDBMS
to outperform an optimized storage format (if you find an example where
it is possible, I'd wager that's only true because the original format
wasn't very well thought-out). It's essentially a given that a given set
of hardware will be able to handle a higher load of storing and
retrieving emails using maildir rather than a database (unless you get
enough messages in a directory that it starts choking the filesystem).
But if you want to do something like search for specific emails, there's
a much better chance that a database will outperform maildir, especially
if you're searching the message body. And there's other potential
applications where a database would outperform maildir as well.
So, in a nutshell, if you're not going to try doing something more
advanced than just storing and retrieving email, it's unlikely that
you'll be happy with storing that email in a database. The further off
that 'beaten path' you get, the more likely you are to see benefit from
using a database.
--
Jim C. Nasby, Database Architect decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828
Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"
Re: The Future of Email is SQL
Posted by qqqq <qq...@usermail.com>.
> I would defer to the smart people to figure out the details. However I do
> wonder if the actual body content of the message would be best stored in a
> file and the SQL used to store anything and everything you would want to
> index. That would keep the SQL file size down if that's an issue. However,
> SQL databases might have to be changed to accomodate the needs to store
> email.
I think this is what I was getting at early in the thread. I would think
that a 5 MB body would do better on file but I don't know enough in regards
to DBs to even make a call.
QQQQ
Re: The Future of Email is SQL
Posted by Marc Perkel <ma...@perkel.com>.
NM Public wrote:
> Sur 2006-06-09, Marc Perkel skribis:
>>
>> Perhaps the headers and other information that you would index be
>> kept in the database and the body of the message stored somewhere
>> else, perhaps even as files.
>
>
> It seems that this is what Zimbra does. Check out my blog post here:
>
> For IMAP, "SQL just sucks"
> <http://deflexion.com/2006/06/for-imap-sql-just-sucks>
>
> especially the comment from KevinH of Zimbra, which includes this:
>
> "It's true that as a mailstore SQL sucks. Databases are not designed
> to store large blobs of data. However in Zimbra's case messages aren't
> stored in the database. Only *meta* data is stored behind a SQL
> interface. [...]"
>
> I hope this is useful. Thanks for inspiring interesting discussion Marc!
> Nancy
> (sent via gmane.mail.spam.spamassassin.general)
>
I would defer to the smart people to figure out the details. However I
do wonder if the actual body content of the message would be best stored
in a file and the SQL used to store anything and everything you would
want to index. That would keep the SQL file size down if that's an
issue. However, SQL databases might have to be changed to accomodate the
needs to store email.
Re: The Future of Email is SQL
Posted by NM Public <ag...@nm.deflexion.com>.
Sur 2006-06-09, Marc Perkel skribis:
>
> Perhaps the headers and other information that you would index
> be kept in the database and the body of the message stored
> somewhere else, perhaps even as files.
It seems that this is what Zimbra does. Check out my blog post
here:
For IMAP, "SQL just sucks"
<http://deflexion.com/2006/06/for-imap-sql-just-sucks>
especially the comment from KevinH of Zimbra, which includes
this:
"It's true that as a mailstore SQL sucks. Databases are not
designed to store large blobs of data. However in Zimbra's case
messages aren't stored in the database. Only *meta* data is
stored behind a SQL interface. [...]"
I hope this is useful. Thanks for inspiring interesting
discussion Marc!
Nancy
(sent via gmane.mail.spam.spamassassin.general)
--
Nancy McGough
Infinite Ink: <http://www.ii.com/>
Bookmarks & Blog: <http://deflexion.com/>
Re: The Future of Email is SQL
Posted by Marc Perkel <ma...@perkel.com>.
Jim C. Nasby wrote:
> On Fri, Jun 09, 2006 at 02:25:52PM -0600, qqqq wrote:
>
>> My point here is - think outside the box. I'm going to be lobbying IMAP
>> server developers to include SQL backends. exim could pipe data into a
>> local delivery agent, or it can have features written to write directly
>> to the SQL backend.
>>
>> Thoughts ..... ?
>>
>> Because I am an SQL dummy, I do have this question. Would aps like Mysql and Postgres be able to handle 10,000+ users with an average of 50 MB of email?
>>
>
> There are people happily running PostgreSQL with terrabyte databases.
> It's really a question of how much concurrency you need.
>
> One nice thing about databases is they make it possible to do things
> like partition your tables by month/week/whatever. You can then move
> older data onto larger partitions that use slower, cheaper drives.
>
Perhaps the headers and other information that you would index be kept
in the database and the body of the message stored somewhere else,
perhaps even as files.
I'm just trying to inspire thought and creativity here btw.
Re: The Future of Email is SQL
Posted by "Jim C. Nasby" <de...@decibel.org>.
On Fri, Jun 09, 2006 at 02:25:52PM -0600, qqqq wrote:
>
> My point here is - think outside the box. I'm going to be lobbying IMAP
> server developers to include SQL backends. exim could pipe data into a
> local delivery agent, or it can have features written to write directly
> to the SQL backend.
>
> Thoughts ..... ?
>
> Because I am an SQL dummy, I do have this question. Would aps like Mysql and Postgres be able to handle 10,000+ users with an average of 50 MB of email?
There are people happily running PostgreSQL with terrabyte databases.
It's really a question of how much concurrency you need.
One nice thing about databases is they make it possible to do things
like partition your tables by month/week/whatever. You can then move
older data onto larger partitions that use slower, cheaper drives.
--
Jim C. Nasby, Database Architect decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828
Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"
Re: The Future of Email is SQL
Posted by qqqq <qq...@usermail.com>.
>>That would be about 500 gigs of email. Fry's Electronics has drives that size on special for $189. So - I'd say yes, should be fairly easy to scale up to that size and beyond.
I believe it would be approx 200 Gigs
QQQQ
Re: The Future of Email is SQL
Posted by Logan Shaw <ls...@emitinc.com>.
On Fri, 9 Jun 2006, Marc Perkel wrote:
> qqqq wrote:
>> Because I am an SQL dummy, I do have this question. Would aps like Mysql
>> and Postgres be able to handle 10,000+ users with an average of 50 MB of
>> email? I really don't know.
>> Also, does the body just get written to a table?
> That would be about 500 gigs of email. Fry's Electronics has drives that size
> on special for $189. So - I'd say yes, should be fairly easy to scale up to
> that size and beyond.
That seems like a red herring considering 500GB of e-mail
still takes about 500GB of disk space whether it's a database
engine or the kernel's filesystem driver writing it to disk.
Yes, there will be differences in storage efficiency, but
they're minor.
And to answer QQQQ's question, yeah, the body would be written
to a table. Because when you store things in a SQL database,
everything you store is written into a table. For what it's
worth, you can store stuff like e-mail in a BLOB (Binary Large
OBject) or a similar type of field that is specifically meant
to be able to handle data of arbitrary length.
- Logan
Re: The Future of Email is SQL
Posted by Jason Marshall <ma...@spots.ab.ca>.
> That would be about 500 gigs of email. Fry's Electronics has drives that size
> on special for $189. So - I'd say yes, should be fairly easy to scale up to
> that size and beyond.
You really think one 500 gig disk is going to give you anywhere close to
the performance you need to accomodate 500 active gigs of mailboxes? If
you had 30x 18 gig fibre-channel drives spread out over many controllers
and many machines you might have half a chance of keeping up.
If you put, say, 5 of those disks on each of 6 servers, you'd still need a
way to (reliably) aggregate those 30 disks into one large storage
facility, and the database engine on all 6 of those servers would have to
be aware that it was just 1/6 of the equation at all times.
There would be no redundancy at that point. You could probably get a 7th
server with a bunch of large slow disks and use that to back up the data
on the real cluster. But you'd never be able to move 'production' over to
the backup server, there'd just be too little disk throughput for it to
work.
And by the way, 10k users isn't a lot. I have 3000 users, and I'd
consider us to be a miniscule operation compared to many others out there.
Scale this to 250k users and we're talking...
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
| Jason Marshall, marshalj@spots.ab.ca. Spots InterConnect, Inc. Calgary, AB |
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Re: The Future of Email is SQL - What drives do you use?
Posted by Jason Marshall <ma...@spots.ab.ca>.
> OK, I'm sorry for changing the subject but I have had good results with 18 and 36 GB IBM SCSI
> drives.
>
> What do you use?
I generally use Seagate. Used to use IBM/Hitachi and Fujitsu. Still
would if they were easier to find in stock around here. Have used
Quantums, and long long ago Micropolis. Both should be avoided...
Found a pile of new 4 gig SCSI disks on ebay, and have been using those
for linux system disks for the last couple years. They're SCSI 2, great
for booting from, they last forever... Ah, remember the good old days
when you could buy a disk that was the right size for the job, not 1400x
bigger than what you need...
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
| Jason Marshall, marshalj@spots.ab.ca. Spots InterConnect, Inc. Calgary, AB |
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Re: The Future of Email is SQL - What drives do you use?
Posted by DAve <da...@pixelhammer.com>.
qqqq wrote:
> | Between two mail gateways and three toasters we have 14 disks that never
> | stop seeking, never, 24/7/365. A consumer grade storage device would
> | scream "mommy" and wet itself.
> |
> | DAve
>
> OK, I'm sorry for changing the subject but I have had good results with 18 and 36 GB IBM SCSI
> drives.
>
> What do you use?
>
Fujitsu is nice, Seagate is better. WD and Maxtor make poor doorstops as
they are too light, but they make a funny plunk sound when they hit water.
But it's the model of the drive and it's intended purpose that matters,
not so much the label. Every manufacturer makes consumer grade equipment.
DAve
--
Three years now I've asked Google why they don't have a
logo change for Memorial Day. Why do they choose to do logos
for other non-international holidays, but nothing for
Veterans?
Maybe they forgot who made that choice possible.
Re: The Future of Email is SQL - What drives do you use?
Posted by qqqq <qq...@usermail.com>.
| Between two mail gateways and three toasters we have 14 disks that never
| stop seeking, never, 24/7/365. A consumer grade storage device would
| scream "mommy" and wet itself.
|
| DAve
OK, I'm sorry for changing the subject but I have had good results with 18 and 36 GB IBM SCSI
drives.
What do you use?
QQQQ
Re: The Future of Email is SQL
Posted by DAve <da...@pixelhammer.com>.
Marc Perkel wrote:
>
>
> qqqq wrote:
>>
>> My point here is - think outside the box. I'm going to be lobbying IMAP
>> server developers to include SQL backends. exim could pipe data into a
>> local delivery agent, or it can have features written to write directly
>> to the SQL backend.
>>
>> Thoughts ..... ?
We are looking at using DBMail as a mail archive for clients to pull
lost and historic mail at a later date. But not for daily use.
>> Because I am an SQL dummy, I do have this question. Would aps like
>> Mysql and Postgres be able to handle 10,000+ users with an average of
>> 50 MB of email?
>> I really don't know.
>>
>> Also, does the body just get written to a table?
>>
>> Enlighten me,
>>
>> QQQQ
> That would be about 500 gigs of email. Fry's Electronics has drives that
> size on special for $189. So - I'd say yes, should be fairly easy to
> scale up to that size and beyond.
If you are building a high performance mail server, even just a
mailstore, you aren't buying hardware at Fry's. Our mail gateways take
in 2gb of messages a day for 6k+ accounts. That is after we have
rejected 70% of the connections.
Between two mail gateways and three toasters we have 14 disks that never
stop seeking, never, 24/7/365. A consumer grade storage device would
scream "mommy" and wet itself.
DAve
--
Three years now I've asked Google why they don't have a
logo change for Memorial Day. Why do they choose to do logos
for other non-international holidays, but nothing for
Veterans?
Maybe they forgot who made that choice possible.
Re: The Future of Email is SQL
Posted by Marc Perkel <ma...@perkel.com>.
qqqq wrote:
>
> My point here is - think outside the box. I'm going to be lobbying IMAP
> server developers to include SQL backends. exim could pipe data into a
> local delivery agent, or it can have features written to write directly
> to the SQL backend.
>
> Thoughts ..... ?
> Because I am an SQL dummy, I do have this question. Would aps like
> Mysql and Postgres be able to handle 10,000+ users with an average of
> 50 MB of email?
>
> I really don't know.
>
> Also, does the body just get written to a table?
>
> Enlighten me,
>
> QQQQ
That would be about 500 gigs of email. Fry's Electronics has drives that
size on special for $189. So - I'd say yes, should be fairly easy to
scale up to that size and beyond.
Re: The Future of Email is SQL
Posted by qqqq <qq...@usermail.com>.
My point here is - think outside the box. I'm going to be lobbying IMAP
server developers to include SQL backends. exim could pipe data into a
local delivery agent, or it can have features written to write directly
to the SQL backend.
Thoughts ..... ?
Because I am an SQL dummy, I do have this question. Would aps like Mysql and Postgres be able to handle 10,000+ users with an average of 50 MB of email?
I really don't know.
Also, does the body just get written to a table?
Enlighten me,
QQQQ
Re: The Future of Email is SQL
Posted by Jay Plesset <ja...@dp-design.com>.
"fast enough" is a value judgement.
Fast enough may be ok, if you have a few hundred or even a few thousand
users, saving small mailboxes.
In a large scale system, where you have a million users, each of which
has thousands of messages, I doubt any current database, SQL or other
will have that kind of performance.
I regularly use a mail server capable of handling that kind of load.
It's free, and will eventually be open sourced. Sun Java System
Messaging Server. Runs on Solaris, Soaris X86, Linux.
Uses individual files for each message.
jay plesset
sr. tech support engineer.
Sun Microsystem.
Marc Perkel wrote:
>After considerable experimenting and thinking things through I thought
>I'd start a thread on the future of email to start planting the seeds of
>where MTA development needs to go. I'm convinced that someday soon we
>will all realize that MBOX and MAILDIR are obsolete technologies and
>that the future is going to be SQL based storage.
>
>First - before everyone starts screaming about speed comparisons, I'm
>not going to go there. Every storage technology has it's advantages and
>disadvantages but I'm just going to say that SQL based mail storage is
>fast enough. The advantages of SQL has to do with power and not with
>speed. Those who would choose it would do so because they want to do new
>things that you can do with a database and can't do without one.
>
>SQL has several advantages. You don't have t deal with the quirks of the
>underlying file system or OS. It takes care of all the locking issues
>and indexing and makes it so that multiple applications can seamlessly
>access the data. With an SQL backend email can be stored from the MTA,
>read from and IMAP client that accesses the same database, and the spam
>filtering engine will have access to the stored email as well.
>
>To give you some examples of what could be done .....
>
>Suppose a spammer sends 1000 phishing spams to your users and then you
>figure out that the 1000 spams already delivered is spam. With a
>database you can do a query to retroactively delete spam that was
>already delivered to the mailboxes. This could also be used to
>retroactively delete viruses already delivered.
>
>Spam filtering programs can lookup existing email in existing folders
>and compare it with new email already deliverd to help determine more
>accurately if a message is spam or not. For example, if the host server
>has a reputation for 100% ham then it can deliver new email without
>running it through Spam Assassin. If programs like Spamassassin can
>access existing email in existing folders it can evaluate new email
>using tricks no one has yet considered.
>
>SQL databases allow for multiple masters and slaves and replication that
>lets you create a cluster that never fails under any conditions. It
>would be far easier to create a system that is always on and always
>backed up.
>
>An SQL backend allows you to use a wide variety of tools, programming
>languages, operating systems in order for you to easily integrate more
>easily than non database systems.
>
>And - this is important - once you have a database then new things that
>no one has yet thought of will be possible and new things we've never
>heard of will be developed because the new power will lend to the
>development of more tricks than you can do without database power.
>
>My point here is - think outside the box. I'm going to be lobbying IMAP
>server developers to include SQL backends. exim could pipe data into a
>local delivery agent, or it can have features written to write directly
>to the SQL backend.
>
>Thoughts ..... ?
>
>
> -- ## List details at http://www.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/ ## Please use the Wiki with
> this list - http://www.exim.org/eximwiki/
>
Re: The Future of Email is SQL
Posted by Marc Perkel <ma...@perkel.com>.
John Rudd wrote:
>
> On Jun 9, 2006, at 1:19 PM, Marc Perkel wrote:
>
>> After considerable experimenting and thinking things through I thought
>> I'd start a thread on the future of email to start planting the seeds of
>> where MTA development needs to go. I'm convinced that someday soon we
>> will all realize that MBOX and MAILDIR are obsolete technologies and
>> that the future is going to be SQL based storage.
>>
>
> Have you looked at dbmail?
>
> IMAP on top of MySQL.
I'm aware of it. What I'm hoping for is a MySQL backend for Dovecot.
Timo has done some work in that direction. I'm hoping that after the 1.0
release that he works on it again. I think I can create some sort of
Exim frontend if Timo adds it to dovecot.
Re: The Future of Email is SQL
Posted by John Rudd <jr...@ucsc.edu>.
On Jun 9, 2006, at 1:19 PM, Marc Perkel wrote:
> After considerable experimenting and thinking things through I thought
> I'd start a thread on the future of email to start planting the seeds
> of
> where MTA development needs to go. I'm convinced that someday soon we
> will all realize that MBOX and MAILDIR are obsolete technologies and
> that the future is going to be SQL based storage.
>
Have you looked at dbmail?
IMAP on top of MySQL.
Re: The Future of Email is SQL
Posted by Marc Perkel <ma...@perkel.com>.
Steve Thomas wrote:
> While this is quite an interesting topic, I have to ask why it's on the
> spamassassin list. Message stores aren't spamassassin specific and this is
> already a pretty high-volume list. Does this discussion really belong
> here?
>
> St-
>
>
>
The reason I posted it here as well as in Dovecot and Exim lists is
because I think the real benefit comes when SA is also integrated into
the system. With a DB in place SA could do things like delete email
that's already been delivered. That way something that was thought to
not be spam but later determined to be spam can be deleted
retroactively. And - I think with the ability to tie into the DB that
new spam detection tricks can be used to identify spam based on what the
user already has.
Picture the MTA, SA, and the IMAP server all integrated into a single
database. The purpose of starting this thread is to inspire thought.
Then maybe down the line this will happen.
Re: The Future of Email is SQL
Posted by Steve Thomas <li...@sthomas.net>.
While this is quite an interesting topic, I have to ask why it's on the
spamassassin list. Message stores aren't spamassassin specific and this is
already a pretty high-volume list. Does this discussion really belong
here?
St-
Re: The Future of Email is SQL
Posted by Marc Perkel <ma...@perkel.com>.
Jim C. Nasby wrote:
> On Fri, Jun 09, 2006 at 06:16:15PM -0400, Rob McEwen wrote:
>
>>>> MS Exchange... one big Database
>>>>
>> Exactly...
>>
>> And that is one reason why I wouldn't touch this SQL idea with a 10 foot
>> pole.. the fact that Exchange works this way only proves my point... I hear
>> all the time about Exchange servers crashing and the administrator having to
>> rebuild the database while the mail server is down for the next 10 hours.
>>
>
> Just because MS couldn't figure out how to do this correctly doesn't
> mean it can't be done.
>
Thanks Jim. The idea here is to be forward looking and only do it if the
databases are up to it. My thinking is that DBs will continue to improve
to the point where using them makes more and more sense. Obviously if
the DB isn't up to the task then there's no readon to do it.
Here's something called DBMail that looks like it's on the right track.
http://www.dbmail.org/dokuwiki/doku.php?id=bigpicture
Re: The Future of Email is SQL
Posted by "Jim C. Nasby" <de...@decibel.org>.
On Fri, Jun 09, 2006 at 06:16:15PM -0400, Rob McEwen wrote:
>
> >>MS Exchange... one big Database
>
> Exactly...
>
> And that is one reason why I wouldn't touch this SQL idea with a 10 foot
> pole.. the fact that Exchange works this way only proves my point... I hear
> all the time about Exchange servers crashing and the administrator having to
> rebuild the database while the mail server is down for the next 10 hours.
Just because MS couldn't figure out how to do this correctly doesn't
mean it can't be done.
--
Jim C. Nasby, Database Architect decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828
Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"
Re: The Future of Email is SQL
Posted by John Rudd <jr...@ucsc.edu>.
On Jun 9, 2006, at 3:16 PM, Rob McEwen wrote:
>
>>> MS Exchange... one big Database
>
> Exactly...
>
> And that is one reason why I wouldn't touch this SQL idea with a 10
> foot
> pole.. the fact that Exchange works this way only proves my point... I
> hear
> all the time about Exchange servers crashing and the administrator
> having to
> rebuild the database while the mail server is down for the next 10
> hours.
>
> The bottom line is that using a SQL DB backend as mail storage is
> putting
> all your eggs in one basket.
>
Not really. With MySQL you can mirror and stripe data across multiple
back end servers. You lose one of the back end storage nodes? No big
deal. Just get a new one up and running before you lose all of its
mirrors.
Though, I would agree that using MS Exchange would be a bad idea.
Re: The Future of Email is SQL
Posted by kbaker <kb...@missionvi.com>.
Mike Jackson wrote:
> I can't recall seeing any mention in this thread of DBmail (dbmail.org),
> which already exists and is an all-in-one SMTP/POP3/IMAP server with
> MySQL or Postgres message storage (with support for SQLite on the way).
> It's been in development for three or four years, and from what I
> remember is used by the developers on a mail system with 100K+ users. I
> like it as a concept, but haven't been brave enough to put it into
> production.
>
Yes I mentioned this the other day. Not too much response though.
DBMail is 5-6 years old. Especially active for the past 4 years. There are a
number of production systems out there; some with thousands of users.
It can be setup all-in-one or integrated with Postfix or other with no problems.
Features such as lmtp, maildrop, and now Sieve support for message filtering,
put it on par with Cyrus from at least a feature standpoint.
I haven't seen benchmarking against the more well known servers, but with an
optimized DB install it performs very well. In addition, it has many great
possibilities with the RDBMS back-end. Clustering, replication not to mention
direct SQL access to the data store which gives a lot of developers a lower
learning curve for extending the software.
It really is worth taking a look. We are using it on an install with around 300
users... with no problems. We are using a dbmail-postfix-mysql-amavis setup,
with mysql replication to a hot backup server for fail-over. Obviously a small
install, but it has been going very well.
- Kevin
Re: For those who are considering a Barracuda Network Device server
Posted by "Michele Neylon :: Blacknight.ie" <mi...@blacknight.ie>.
Jeff Chan wrote:
>
> Doesn't Barracuda use SpamAssassin in their boxes? If so it's
> not too surprising that it wouldn't perform much differently from
> SpamAssassin.... :-)
It's probably using an old version.... *shrug*
>
> Barracuda may not use SARE, so SARE may indeed be better.
Quite possibly
--
Mr Michele Neylon
Blacknight Solutions
Quality Business Hosting & Colocation
http://www.blacknight.ie/
Tel. 1850 927 280
Intl. +353 (0) 59 9183072
Direct Dial: +353 (0)59 9183090
Fax. +353 (0) 59 9164239
Re: For those who are considering a Barracuda Network Device server
Posted by Jeff Chan <je...@surbl.org>.
On Monday, June 12, 2006, 10:23:20 AM, qqqq qqqq wrote:
> I bought a Barracuda Model 400 last October. My current setup is as follows:
Barracuda GW --->> Internal servers ---> Spamassassin server ---> Quarantine or local delivery.
> Although there was a small percentage of spam being caught by adding the Barracuda, this was because
> I added my own Regex rules on the Barracuda. Without my Regex rules, using their "intent" RBL, a
> trained bayes, and SBL-XBL RBL, the devise gave me nothing in terms of more spam captured than
> Spamassassin with SARE. In fact, I don't have concrete numbers but I am willing to put $100 and say
> SA/SARE does better.
Doesn't Barracuda use SpamAssassin in their boxes? If so it's
not too surprising that it wouldn't perform much differently from
SpamAssassin.... :-)
Barracuda may not use SARE, so SARE may indeed be better.
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
Re: For those who are considering a Barracuda Network Device server
Posted by Craig White <cr...@azapple.com>.
On Mon, 2006-06-12 at 19:34 -0700, jdow wrote:
> If I was feeling stinky I'd note that I do not like web administration
> tools as much as I like editing the files myself by hand doing things
> I understand from an overdose of RTFM. And I'm not a Linux guy last
> time I checked myself in front of a mirror.
>
> {^,-} But I'm not. (Besides "ix guy" is perhaps more to the point.
> I also "dabble" with FreeBSD; but, I don't use it for anything
> important yet.) (It's been a contentious day on several lists.
> Some humor was needed.)
----
You mean calling GPL License 'nonsense' wasn't your best effort of the
day?
You hurled similar bombshells on other lists?
Craig
Re: For those who are considering a Barracuda Network Device server
Posted by jdow <jd...@earthlink.net>.
If I was feeling stinky I'd note that I do not like web administration
tools as much as I like editing the files myself by hand doing things
I understand from an overdose of RTFM. And I'm not a Linux guy last
time I checked myself in front of a mirror.
{^,-} But I'm not. (Besides "ix guy" is perhaps more to the point.
I also "dabble" with FreeBSD; but, I don't use it for anything
important yet.) (It's been a contentious day on several lists.
Some humor was needed.)
----- Original Message -----
From: "Brent Kennedy" <br...@cfl.rr.com>
>I took a look at them as a way to possibly go to a gui spam server because
> some of the other admins at my company are not linux gurus by any stretch,
> but these lacked some of the necessary functionality that would give me
> cause to actually pay for one. Course.. If anyone doesn't know.. Use
> webmin, it's a great alternative to doing things via command line...
> Especially if your not a linux guy like most of this list. Need a secure
> connection, just use the webmin ssl feature.
>
> :)
>
> -Brent
>
> -----Original Message-----
> From: L. Mark Stone [mailto:lmstone@rnome.com]
>
> On Mon, 2006-06-12 at 11:41 -0600, qqqq wrote:
>> | I pretty much at this time strictly use the Barracuda as a buffer to
>> | 'tone' down traffic that would make our server drop to its knees. We
>> | are in process of getting a firewall in place and when that happens,
>> | the Barracuda will probably go bye..bye when I start building access
> lists.
>>
>> That's exactly what mine was doing. It allowed for me to run SA/SARE with
> a less beefy box.
>>
>> B
>
> Our SonicWall PRO2040 firewall does RBL checking, which was a pleasant
> surprise to us since that feature was not a decision criterium for us.
>
> But, using the RBL feature has reduced the load on our Postfix/SA box
> considerably--and for much less $$$ than a Barracuda.
>
>
>
>
> --
> _________________________________________________________
> A Message From... L. Mark Stone
>
> Reliable Networks of Maine, LLC
>
> "We manage your network so you can manage your business"
>
> 477 Congress Street
> Portland, ME 04101
> Tel: (207) 772-5678
> Web: http://www.rnome.com
>
> This email was sent from Reliable Networks of Maine LLC.
> It may contain information that is privileged and confidential.
> If you suspect that you were not intended to receive it, please delete it
> and notify us as soon as possible. Thank you.
>
>
RE: For those who are considering a Barracuda Network Device server
Posted by Brent Kennedy <br...@cfl.rr.com>.
I took a look at them as a way to possibly go to a gui spam server because
some of the other admins at my company are not linux gurus by any stretch,
but these lacked some of the necessary functionality that would give me
cause to actually pay for one. Course.. If anyone doesn't know.. Use
webmin, it's a great alternative to doing things via command line...
Especially if your not a linux guy like most of this list. Need a secure
connection, just use the webmin ssl feature.
:)
-Brent
-----Original Message-----
From: L. Mark Stone [mailto:lmstone@rnome.com]
Sent: Monday, June 12, 2006 8:44 PM
To: users@spamassassin.apache.org
Subject: Re: For those who are considering a Barracuda Network Device server
On Mon, 2006-06-12 at 11:41 -0600, qqqq wrote:
> | I pretty much at this time strictly use the Barracuda as a buffer to
> | 'tone' down traffic that would make our server drop to its knees. We
> | are in process of getting a firewall in place and when that happens,
> | the Barracuda will probably go bye..bye when I start building access
lists.
>
> That's exactly what mine was doing. It allowed for me to run SA/SARE with
a less beefy box.
>
> B
Our SonicWall PRO2040 firewall does RBL checking, which was a pleasant
surprise to us since that feature was not a decision criterium for us.
But, using the RBL feature has reduced the load on our Postfix/SA box
considerably--and for much less $$$ than a Barracuda.
--
_________________________________________________________
A Message From... L. Mark Stone
Reliable Networks of Maine, LLC
"We manage your network so you can manage your business"
477 Congress Street
Portland, ME 04101
Tel: (207) 772-5678
Web: http://www.rnome.com
This email was sent from Reliable Networks of Maine LLC.
It may contain information that is privileged and confidential.
If you suspect that you were not intended to receive it, please delete it
and notify us as soon as possible. Thank you.
Re: For those who are considering a Barracuda Network Device server
Posted by "L. Mark Stone" <lm...@rnome.com>.
On Mon, 2006-06-12 at 11:41 -0600, qqqq wrote:
> | I pretty much at this time strictly use the Barracuda as a buffer to 'tone' down
> | traffic that would make our server drop to its knees. We are in process
> | of getting a firewall in place and when that happens, the Barracuda will
> | probably go bye..bye when I start building access lists.
>
> That's exactly what mine was doing. It allowed for me to run SA/SARE with a less beefy box.
>
> B
Our SonicWall PRO2040 firewall does RBL checking, which was a pleasant
surprise to us since that feature was not a decision criterium for us.
But, using the RBL feature has reduced the load on our Postfix/SA box
considerably--and for much less $$$ than a Barracuda.
--
_________________________________________________________
A Message From... L. Mark Stone
Reliable Networks of Maine, LLC
"We manage your network so you can manage your business"
477 Congress Street
Portland, ME 04101
Tel: (207) 772-5678
Web: http://www.rnome.com
This email was sent from Reliable Networks of Maine LLC.
It may contain information that is privileged and confidential.
If you suspect that you were not intended to receive it, please
delete it and notify us as soon as possible. Thank you.
Re: For those who are considering a Barracuda Network Device server
Posted by qqqq <qq...@usermail.com>.
| I pretty much at this time strictly use the Barracuda as a buffer to 'tone' down
| traffic that would make our server drop to its knees. We are in process
| of getting a firewall in place and when that happens, the Barracuda will
| probably go bye..bye when I start building access lists.
That's exactly what mine was doing. It allowed for me to run SA/SARE with a less beefy box.
B
Re: For those who are considering a Barracuda Network Device server
Posted by Duane Hill <d....@yournetplus.com>.
We have a 400 as well. I don't think one can even compare SA
out-of-the-box and the Barracuda. I'm catching more Spam with the use of
SA with no rules loaded than what our Barracuda is tagging. I've taken
messages that came off the Barracuda and thew it through SA. SA scored
almost 2.5 points higher than the Barracuda in several cases. I pretty
much at this time strictly use the Barracuda as a buffer to 'tone' down
traffic that would make our server drop to its knees. We are in process
of getting a firewall in place and when that happens, the Barracuda will
probably go bye..bye when I start building access lists.
On Mon, 12 Jun 2006, qqqq wrote:
> All,
>
> I bought a Barracuda Model 400 last October. My current setup is as follows:
> Barracuda GW ---> Internal servers ---> Spamassassin server ---> Quarantine or local delivery.
>
> Although there was a small percentage of spam being caught by adding the Barracuda, this was because
> I added my own Regex rules on the Barracuda. Without my Regex rules, using their "intent" RBL, a
> trained bayes, and SBL-XBL RBL, the devise gave me nothing in terms of more spam captured than
> Spamassassin with SARE. In fact, I don't have concrete numbers but I am willing to put $100 and say
> SA/SARE does better.
>
> Because I am a customer, I have access to the Barracuda Networks forums. I am not the only one
> unimpressed and since it's a moderated forum, all the postings I have made which have a negative
> undertones, do not get posted.
>
> OK, enough of that. I bought the server in October (8 months ago) for about $4800. This included a
> years license. Well, the server has a bad drive based on their support's diagnostics. They want me
> to pay $899 to have it replaced! FWIW, if you open the box yourself, it voids support.
>
> I want to say that if your thinking of buying one of the Barracuda Networks Spam Firewall
> (www.barracudanetworks.com), save your money!. I would have been better off spending about $2000
> for two decent servers and running SA/SARE at the perimeter.
>
> I hope this helps somebody else from making the same mistake.
>
> Brian
>
--
"This message was sent using 100% recycled electrons."
RE: For those who are considering a Barracuda Network Device server
Posted by Greg Allen <sa...@floridacpu.com>.
>
> I hope this helps somebody else from making the same mistake.
>
> Brian
>
I looked all over the Internet about 5 years ago for the best anti-spam
software. I wanted it to be Windows-based easy to use, etc.
Well, I found the best anti-spam software and it turned out to be
SpamAssassin. It wasn't Windows based, and wasn't necessarily easy to learn
and use. But I'm glad it was created (mostly) for Linux/Unix since those
operating systems are much more robust as front end email servers, and I'm
glad I took the time to learn SA.
Any of these anti-spam vendors like Barracuda are just bad imitations of SA.
Some not as bad as others, but none are as good as SA.
Regarding Barracuda Networks... I hate their radio commercials... what BS
hype artists. They make it sound like they are better than anything
available for anti-spam and anti-virus and that you will never get another
spam or virus if you buy their product.
4,800 for a Barracuda? lol No thanks...
I hope someone sues them for deceptive advertising to shut down those
annoying commercials.
For those who are considering a Barracuda Network Device server
Posted by qqqq <qq...@usermail.com>.
All,
I bought a Barracuda Model 400 last October. My current setup is as follows:
Barracuda GW ---> Internal servers ---> Spamassassin server ---> Quarantine or local delivery.
Although there was a small percentage of spam being caught by adding the Barracuda, this was because
I added my own Regex rules on the Barracuda. Without my Regex rules, using their "intent" RBL, a
trained bayes, and SBL-XBL RBL, the devise gave me nothing in terms of more spam captured than
Spamassassin with SARE. In fact, I don't have concrete numbers but I am willing to put $100 and say
SA/SARE does better.
Because I am a customer, I have access to the Barracuda Networks forums. I am not the only one
unimpressed and since it's a moderated forum, all the postings I have made which have a negative
undertones, do not get posted.
OK, enough of that. I bought the server in October (8 months ago) for about $4800. This included a
years license. Well, the server has a bad drive based on their support's diagnostics. They want me
to pay $899 to have it replaced! FWIW, if you open the box yourself, it voids support.
I want to say that if your thinking of buying one of the Barracuda Networks Spam Firewall
(www.barracudanetworks.com), save your money!. I would have been better off spending about $2000
for two decent servers and running SA/SARE at the perimeter.
I hope this helps somebody else from making the same mistake.
Brian
Re: The Future of Email is SQL
Posted by Mike Jackson <mj...@mightymerchant.com>.
I can't recall seeing any mention in this thread of DBmail (dbmail.org),
which already exists and is an all-in-one SMTP/POP3/IMAP server with MySQL
or Postgres message storage (with support for SQLite on the way). It's been
in development for three or four years, and from what I remember is used by
the developers on a mail system with 100K+ users. I like it as a concept,
but haven't been brave enough to put it into production.
RE: The Future of Email is SQL
Posted by Martin Hepworth <ma...@solid-state-logic.com>.
Well yes Exchange does have it's problems (its much better than it used to
be), but ya gotta remember the underlying "DB" is Access.
I think there are moves afoot for the next version of MS-Ex to be able to
run with SQl-Server as the backend datastore (2003 may already have this
ability) which is v. usefull for large (1000+) user bases.
Kinda proves the point really, you need a proper DB for this sort of thing
not some tin pot 'user' thing.
--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
> -----Original Message-----
> From: Rob McEwen [mailto:rob@powerviewsystems.com]
> Sent: 09 June 2006 23:16
> To: users@spamassassin.apache.org
> Subject: RE: The Future of Email is SQL
>
>
> >>MS Exchange... one big Database
>
> Exactly...
>
> And that is one reason why I wouldn't touch this SQL idea with a 10 foot
> pole.. the fact that Exchange works this way only proves my point... I
> hear
> all the time about Exchange servers crashing and the administrator having
> to
> rebuild the database while the mail server is down for the next 10 hours.
>
> The bottom line is that using a SQL DB backend as mail storage is putting
> all your eggs in one basket.
>
> I have a much simpler solution to accomplish the problem that this was
> idea
> was originally attempting to solve... simply place the spams that are
> caught
> in a folder on the mail server that is accessible via webmail. Then create
> a
> separate program to periodically enumerate through the spam folder in all
> the accounts on the server to delete spams over X days old.
>
> If needed, you could still have a database with the basic info about the
> spams (date received, subject line, recipients, from, message file name,
> etc) to use for e-mailing "digests" to the user... and this DB's stability
> wouldn't then have to be tied to the overall reliability/stability of mail
> services.
>
> Also keep in mind that SQL doesn't always mean better performance... I've
> seen many web sites that deliver content dynamically from a SQL database
> backend where there were noticeably large delays between page loads, for
> example.
>
> Rob McEwen
> PowerView Systems
> rob@PowerViewSystems.com
>
**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.
**********************************************************************
RE: The Future of Email is SQL
Posted by "John D. Hardin" <jh...@impsec.org>.
On Fri, 9 Jun 2006, Rob McEwen wrote:
> >>MS Exchange... one big Database
>
> Exactly...
>
> And that is one reason why I wouldn't touch this SQL idea with a
> 10 foot pole.. the fact that Exchange works this way only proves
> my point... I hear all the time about Exchange servers crashing
> and the administrator having to rebuild the database while the
> mail server is down for the next 10 hours.
>
> The bottom line is that using a SQL DB backend as mail storage is
> putting all your eggs in one basket.
Not to mention you get the same problem that everyone complains about
with the Windows Registry: everything is buried in this black box
storage format that you can only access with specific tools - you lose
the ability to access and process email messages with the rich suite
of simple text processing tools that are available, and the ability to
read your email with something as simple as a text editor.
Granted, there is less of the "impenetrable black box" situation with
a SQL database than there is with the Registry, but the same concepts
and limitations apply.
--
John Hardin KA7OHZ ICQ#15735746 http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 - 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The first time I saw a bagpipe, I thought the player was torturing
an octopus. I was amazed they could scream so loudly.
-- cat_herder_5263 on Y! SCOX
-----------------------------------------------------------------------
9 days until SWMBO's Birthday
Re: The Future of Email is SQL
Posted by Marc Perkel <ma...@perkel.com>.
Greg Allen wrote:
>
>> -----Original Message-----
>> From: Rob McEwen [mailto:rob@powerviewsystems.com]
>> Sent: Friday, June 09, 2006 6:16 PM
>> To: users@spamassassin.apache.org
>> Subject: RE: The Future of Email is SQL
>>
>>
>>
>>
>>>> MS Exchange... one big Database
>>>>
>> Exactly...
>>
>> And that is one reason why I wouldn't touch this SQL idea with a 10 foot
>> pole.. the fact that Exchange works this way only proves my
>> point... I hear
>> all the time about Exchange servers crashing and the
>> administrator having to
>> rebuild the database while the mail server is down for the next 10 hours.
>>
>
> Yup, I have worked on Exchange servers for years. 5.5 blew up all the time.
> 2000 not so much at all. I expect 2003 is failrly stable. But regardless, if
> it does go... that group of users is down all day. I know some orgs are
> using clusters on Exchange to help with that problem... but now you have a
> cluster that only one guy knows how to work on. The guy who set it up. So,
> if the cluster gets screwed somehow, you have to find the guy who set it up,
> you then have to fix the cluster, and then the Exchange. Just shoot yourself
> in the head and save some time.
>
> I would rather use Exchange with seperate PST files for each user, and it
> will let you do that. The reason most companies end up on the single HUGE
> database it because Exchange requires that be down to share appointments,
> tasks, etc.
>
>
What I have in mind would be a far better database than Exchange. I'm assuming a really good database like MySQL or Oracle and I'm assuming that in the future that databases will get even better. Spamassassin has switched from DB files to MySQL and I think that kind of evolution will continue. So - I'm trying to plant that idea of a SQL future.
RE: The Future of Email is SQL
Posted by Greg Allen <sa...@floridacpu.com>.
> -----Original Message-----
> From: Rob McEwen [mailto:rob@powerviewsystems.com]
> Sent: Friday, June 09, 2006 6:16 PM
> To: users@spamassassin.apache.org
> Subject: RE: The Future of Email is SQL
>
>
>
> >>MS Exchange... one big Database
>
> Exactly...
>
> And that is one reason why I wouldn't touch this SQL idea with a 10 foot
> pole.. the fact that Exchange works this way only proves my
> point... I hear
> all the time about Exchange servers crashing and the
> administrator having to
> rebuild the database while the mail server is down for the next 10 hours.
Yup, I have worked on Exchange servers for years. 5.5 blew up all the time.
2000 not so much at all. I expect 2003 is failrly stable. But regardless, if
it does go... that group of users is down all day. I know some orgs are
using clusters on Exchange to help with that problem... but now you have a
cluster that only one guy knows how to work on. The guy who set it up. So,
if the cluster gets screwed somehow, you have to find the guy who set it up,
you then have to fix the cluster, and then the Exchange. Just shoot yourself
in the head and save some time.
I would rather use Exchange with seperate PST files for each user, and it
will let you do that. The reason most companies end up on the single HUGE
database it because Exchange requires that be down to share appointments,
tasks, etc.
RE: The Future of Email is SQL
Posted by Rob McEwen <ro...@powerviewsystems.com>.
>>MS Exchange... one big Database
Exactly...
And that is one reason why I wouldn't touch this SQL idea with a 10 foot
pole.. the fact that Exchange works this way only proves my point... I hear
all the time about Exchange servers crashing and the administrator having to
rebuild the database while the mail server is down for the next 10 hours.
The bottom line is that using a SQL DB backend as mail storage is putting
all your eggs in one basket.
I have a much simpler solution to accomplish the problem that this was idea
was originally attempting to solve... simply place the spams that are caught
in a folder on the mail server that is accessible via webmail. Then create a
separate program to periodically enumerate through the spam folder in all
the accounts on the server to delete spams over X days old.
If needed, you could still have a database with the basic info about the
spams (date received, subject line, recipients, from, message file name,
etc) to use for e-mailing "digests" to the user... and this DB's stability
wouldn't then have to be tied to the overall reliability/stability of mail
services.
Also keep in mind that SQL doesn't always mean better performance... I've
seen many web sites that deliver content dynamically from a SQL database
backend where there were noticeably large delays between page loads, for
example.
Rob McEwen
PowerView Systems
rob@PowerViewSystems.com
RE: The Future of Email is SQL
Posted by Greg Allen <sa...@floridacpu.com>.
-----Original Message-----
From: Marc Perkel [mailto:marc@perkel.com]
Sent: Friday, June 09, 2006 4:19 PM
To: users@spamassassin.apache.org
Subject: The Future of Email is SQL
Thoughts ..... ?
-----------------------------------------
MS Exchange... one big Database
You can use Exmerge to do some of what you are looking to do (delete one
email to all users, export dates of email, etc.).
If someone sends 1,000 copies of the same email to all of the users on the
same organization (cc,bcc) the message is stored only once, with pointers to
it.
It may not do everything you are looking to do, not sure.
Just pointing out that Microsoft has already started down that route to some
extent and they may end up using SQL even.