Couch as a mail store?

[Kenneth Kalmer <ke...@gmail.com>]

Hi everyone

This is my first post, please be gentle as I risk ridicule. I've been
lurking here for several months now, learning from others. Disclaimer, I
have yet do do more with couch than updating and running the tests.

How would couch fair as a backend for a mail delivery system (in concept)?
Considering you need high availability and very fast IO. Documents (email
messages) will be created and deleted very often, some almost
instantaneously.

Couch has some great attributes that makes it sound worth exploring further:

* Fast lookup of documents
* Awesome replication for business continuity (especially in a low-latency
environment like GIG-E)
* Scales horizontally
* Ability to pull entire mailbox for user as one result, or at least bundle
X emails together in one response

I can't recall seeing any thread on here in recent history discussing high
document deletion rates, which is effectively the case when people pop their
mail.

Normal filesystem-based storage of mail has other issues:

* Messages often smaller than ethernet jumbo frames, so limited throughput
(couch can overcome this by bundling messages in a single response)
* Mostly limited by disk IO and clever tricks around solid state drive usage
or stripping excessively fast disks

Lets assume nothing about existing mail stores, except that filesystem ones
don't scale will, and I don't even want to consider the possibility of
raping an RDBMS for this.

Everything is exploratory, the thought just crept into my mind a couple of
days ago and I'd like to bounce the idea around with everyone for fun.

Thanks for all the hard work, and everyones patience with newbies and
attackers alike.

Best

-- 
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za

Re: Couch as a mail store?

[Jan Lehnardt <ja...@apache.org>]

On 12 Feb 2009, at 16:51, Kenneth Kalmer wrote:
>
>> A deletion is effectively a set-deleted-flag operation. Compaction  
>> then
>> takes care of getting rid of the file.

> So taken from other threads, you're effectively tasked with running
> compaction outside of your peak time. This is a no brainer if the  
> other
> benefits are in reachable.

Right.


> While I'm here, can the docs still be recovered before compaction?  
> Why I ask
> is that it would be a bonus to be able to access the mails and do some
> statistical reporting before compacting the database, if not, no  
> issues.
> Mail server admins (and those footing the bills) love excessive  
> reporting...

They can, but you're still not advised to do it. If you need any data,  
it should be
in the latest version of a document. Reports could be run on the side  
and stored
into a secondary database.

>> Hmm, not too much information. Let's see, if you have any more  
>> specific
>> questions, just send a follow up :)
>>
>
> Well, lets try and keep this as close as couch as we can and not  
> wander off
> into the nasty world of email systems (except for effectively CRUD-ing
> messages).
>
> So mail arrives at our SMTP server. What would give us the best  
> performance
> for ingesting mail, directly writing each doc as it arrives, or  
> having small
> queues that empty out every X messages / Y seconds (whichever comes  
> first)?
> Considering one of our mail clouds does about 15GB an hour during  
> office
> hours. I know this size isn't anything when you consider larger  
> providers,
> but we're growing constantly and some time in the future we're gonna  
> have to
> become creative in how we store mail.

Single doc inserts get batch-witten to disk each second. This might  
work for your
data and memory requirements. But you can fine-tune that with custom  
queueing
and bulk doc inserts.


> Retrieving mail also becomes interesting, we can use one view to get  
> the
> total number of messages for the mailbox, and then another (with  
> parameters)
> to batch them from couchdb as we deliver them to the client. Would  
> bulk
> updates here be the cheapest way of "mark all as read" or "delete",  
> or would
> you again handle documents individually?

You can set the _deleted member in a bulk-update operation to delete a  
bunch
of docs in one go.

Cheers
Jan
--

Re: Couch as a mail store?

[Paul Davis <pa...@gmail.com>]

On Thu, Feb 12, 2009 at 10:51 AM, Kenneth Kalmer
<ke...@gmail.com> wrote:
> On Thu, Feb 12, 2009 at 4:26 PM, Jan Lehnardt <ja...@apache.org> wrote:
>
>>
>> On 12 Feb 2009, at 15:13, Kenneth Kalmer wrote:
>>
>>  Hi everyone
>>>
>>> This is my first post, please be gentle as I risk ridicule. I've been
>>> lurking here for several months now, learning from others. Disclaimer, I
>>> have yet do do more with couch than updating and running the tests.
>>>
>>
>> No worries, we don't bite (...usually :).
>>
>
> I've noticed :)
>
>  How would couch fair as a backend for a mail delivery system (in concept)?
>>
>
> Two words: Perfect match.
>>
>
> My reason for persuing the concept.
>
>  Considering you need high availability and very fast IO. Documents (email
>> messages) will be created and deleted very often, some almost
>> instantaneously.
>>
>> Couch has some great attributes that makes it sound worth exploring
>> further:
>>
>> * Fast lookup of documents
>> * Awesome replication for business continuity (especially in a low-latency
>> environment like GIG-E)
>> * Scales horizontally
>> * Ability to pull entire mailbox for user as one result, or at least bundle
>> X emails together in one response
>>
>> I can't recall seeing any thread on here in recent history discussing high
>> document deletion rates, which is effectively the case when people pop
>> their
>> mail.
>>
>
> A deletion is effectively a set-deleted-flag operation. Compaction then
>> takes
>> care of getting rid of the file.
>>
>
> So taken from other threads, you're effectively tasked with running
> compaction outside of your peak time. This is a no brainer if the other
> benefits are in reachable.
>
> While I'm here, can the docs still be recovered before compaction? Why I ask
> is that it would be a bonus to be able to access the mails and do some
> statistical reporting before compacting the database, if not, no issues.
> Mail server admins (and those footing the bills) love excessive reporting...
>

Old versions of docs can be recovered exactly until you compact. Also,
in terms of statistical reporting, check out Jan's stats additions
that will help reporting on the internals of CouchDB itself (think #
of requests, time per request, etc).

>
>  Normal filesystem-based storage of mail has other issues:
>>
>> * Messages often smaller than ethernet jumbo frames, so limited throughput
>> (couch can overcome this by bundling messages in a single response)
>> * Mostly limited by disk IO and clever tricks around solid state drive
>> usage
>> or stripping excessively fast disks
>>
>> Lets assume nothing about existing mail stores, except that filesystem ones
>> don't scale will, and I don't even want to consider the possibility of
>> raping an RDBMS for this.
>>
>> Everything is exploratory, the thought just crept into my mind a couple of
>> days ago and I'd like to bounce the idea around with everyone for fun.
>>
>> Thanks for all the hard work, and everyones patience with newbies and
>> attackers alike.
>>
>
> Hey, thanks for the nice words :)
>>
>> Hmm, not too much information. Let's see, if you have any more specific
>> questions, just send a follow up :)
>>
>
> Well, lets try and keep this as close as couch as we can and not wander off
> into the nasty world of email systems (except for effectively CRUD-ing
> messages).
>
> So mail arrives at our SMTP server. What would give us the best performance
> for ingesting mail, directly writing each doc as it arrives, or having small
> queues that empty out every X messages / Y seconds (whichever comes first)?
> Considering one of our mail clouds does about 15GB an hour during office
> hours. I know this size isn't anything when you consider larger providers,
> but we're growing constantly and some time in the future we're gonna have to
> become creative in how we store mail.
>

Using _bulk_docs gives you a direct RAM vs. Speed trade off. The
bigger you can make single inserts the more efficient the entire
system will be. Obviously you'll have to balance that with latency
concerns, but at 15 GiB an hour I'd imagine you'll be hitting RAM
limits before latency is a factor.

> Retrieving mail also becomes interesting, we can use one view to get the
> total number of messages for the mailbox, and then another (with parameters)
> to batch them from couchdb as we deliver them to the client. Would bulk
> updates here be the cheapest way of "mark all as read" or "delete", or would
> you again handle documents individually?
>
> Best
>
>
> --
> Kenneth Kalmer
> kenneth.kalmer@gmail.com
> http://opensourcery.co.za
>

Remember to report any numbers you find back to the list. We like to
have real world examples to point at to give new people a feeling for
what type of stuff they can throw at CouchDB. And it helps with
bragging too. :D

HTH,
Paul Davis

Re: Couch as a mail store?

[Kenneth Kalmer <ke...@gmail.com>]

On Thu, Feb 12, 2009 at 4:26 PM, Jan Lehnardt <ja...@apache.org> wrote:

>
> On 12 Feb 2009, at 15:13, Kenneth Kalmer wrote:
>
>  Hi everyone
>>
>> This is my first post, please be gentle as I risk ridicule. I've been
>> lurking here for several months now, learning from others. Disclaimer, I
>> have yet do do more with couch than updating and running the tests.
>>
>
> No worries, we don't bite (...usually :).
>

I've noticed :)

 How would couch fair as a backend for a mail delivery system (in concept)?
>

Two words: Perfect match.
>

My reason for persuing the concept.

 Considering you need high availability and very fast IO. Documents (email
> messages) will be created and deleted very often, some almost
> instantaneously.
>
> Couch has some great attributes that makes it sound worth exploring
> further:
>
> * Fast lookup of documents
> * Awesome replication for business continuity (especially in a low-latency
> environment like GIG-E)
> * Scales horizontally
> * Ability to pull entire mailbox for user as one result, or at least bundle
> X emails together in one response
>
> I can't recall seeing any thread on here in recent history discussing high
> document deletion rates, which is effectively the case when people pop
> their
> mail.
>

A deletion is effectively a set-deleted-flag operation. Compaction then
> takes
> care of getting rid of the file.
>

So taken from other threads, you're effectively tasked with running
compaction outside of your peak time. This is a no brainer if the other
benefits are in reachable.

While I'm here, can the docs still be recovered before compaction? Why I ask
is that it would be a bonus to be able to access the mails and do some
statistical reporting before compacting the database, if not, no issues.
Mail server admins (and those footing the bills) love excessive reporting...


 Normal filesystem-based storage of mail has other issues:
>
> * Messages often smaller than ethernet jumbo frames, so limited throughput
> (couch can overcome this by bundling messages in a single response)
> * Mostly limited by disk IO and clever tricks around solid state drive
> usage
> or stripping excessively fast disks
>
> Lets assume nothing about existing mail stores, except that filesystem ones
> don't scale will, and I don't even want to consider the possibility of
> raping an RDBMS for this.
>
> Everything is exploratory, the thought just crept into my mind a couple of
> days ago and I'd like to bounce the idea around with everyone for fun.
>
> Thanks for all the hard work, and everyones patience with newbies and
> attackers alike.
>

Hey, thanks for the nice words :)
>
> Hmm, not too much information. Let's see, if you have any more specific
> questions, just send a follow up :)
>

Well, lets try and keep this as close as couch as we can and not wander off
into the nasty world of email systems (except for effectively CRUD-ing
messages).

So mail arrives at our SMTP server. What would give us the best performance
for ingesting mail, directly writing each doc as it arrives, or having small
queues that empty out every X messages / Y seconds (whichever comes first)?
Considering one of our mail clouds does about 15GB an hour during office
hours. I know this size isn't anything when you consider larger providers,
but we're growing constantly and some time in the future we're gonna have to
become creative in how we store mail.

Retrieving mail also becomes interesting, we can use one view to get the
total number of messages for the mailbox, and then another (with parameters)
to batch them from couchdb as we deliver them to the client. Would bulk
updates here be the cheapest way of "mark all as read" or "delete", or would
you again handle documents individually?

Best


-- 
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za

Re: Couch as a mail store?

[Jan Lehnardt <ja...@apache.org>]

On 12 Feb 2009, at 15:13, Kenneth Kalmer wrote:

> Hi everyone
>
> This is my first post, please be gentle as I risk ridicule. I've been
> lurking here for several months now, learning from others.  
> Disclaimer, I
> have yet do do more with couch than updating and running the tests.

No worries, we don't bite (...usually :).


> How would couch fair as a backend for a mail delivery system (in  
> concept)?

Two words: Perfect match.


> Considering you need high availability and very fast IO. Documents  
> (email
> messages) will be created and deleted very often, some almost
> instantaneously.
>
> Couch has some great attributes that makes it sound worth exploring  
> further:
>
> * Fast lookup of documents
> * Awesome replication for business continuity (especially in a low- 
> latency
> environment like GIG-E)
> * Scales horizontally
> * Ability to pull entire mailbox for user as one result, or at least  
> bundle
> X emails together in one response
>
> I can't recall seeing any thread on here in recent history  
> discussing high
> document deletion rates, which is effectively the case when people  
> pop their
> mail.

A deletion is effectively a set-deleted-flag operation. Compaction  
then takes
care of getting rid of the file.


> Normal filesystem-based storage of mail has other issues:
>
> * Messages often smaller than ethernet jumbo frames, so limited  
> throughput
> (couch can overcome this by bundling messages in a single response)
> * Mostly limited by disk IO and clever tricks around solid state  
> drive usage
> or stripping excessively fast disks
>
> Lets assume nothing about existing mail stores, except that  
> filesystem ones
> don't scale will, and I don't even want to consider the possibility of
> raping an RDBMS for this.
>
> Everything is exploratory, the thought just crept into my mind a  
> couple of
> days ago and I'd like to bounce the idea around with everyone for fun.
>
> Thanks for all the hard work, and everyones patience with newbies and
> attackers alike.

Hey, thanks for the nice words :)

Hmm, not too much information. Let's see, if you have any more specific
questions, just send a follow up :)

Cheers
Jan
--

Re: [user] Couch as a mail store?

[Kenneth Kalmer <ke...@gmail.com>]

On Thu, Feb 12, 2009 at 4:59 PM, Jan Lehnardt <ja...@apache.org> wrote:

>
> On 12 Feb 2009, at 15:50, Wout Mertens wrote:
>
>  On Feb 12, 2009, at 3:13 PM, Kenneth Kalmer wrote:
>>
>>  Normal filesystem-based storage of mail has other issues:
>>>
>>> * Messages often smaller than ethernet jumbo frames, so limited
>>> throughput
>>> (couch can overcome this by bundling messages in a single response)
>>>
>>
>> wow that's really specialized. Is that really an issue?
>
>
If you're using NFS or GFS, yes, it becomes any issue. We had discussions
with EMC (storage company) a while ago, and one of their supreme techies
(not used lightly, got his doctorate and contributes to the NFS project)
admitted that no matter how great their products are, they're useless for
mail systems where the messages are stored in individual files...


>>  * Mostly limited by disk IO and clever tricks around solid state drive
>>> usage
>>> or stripping excessively fast disks
>>>
>>
>> Hmmm makes me wonder - CouchDB doesn't change the position of received
>> documents on disk, so retrieving a view could mean many disk accesses.
>>
>> Is that correct? Is that something all databases suffer from? I suppose
>> the only answer for that is caching?
>>
>
> The file and b-tree structure is actually made to require a minimal
> number of disk head seeks (which is what makes disks slow) and
> in a typical usage scenario, upper nodes of the btree are cached
> in the filesystem buffer cache too, so there's another speedup.
>

This is great news. I'll pickup on your initial reply in a second.

-- 
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za

Re: [user] Couch as a mail store?

[Jan Lehnardt <ja...@apache.org>]

On 12 Feb 2009, at 15:50, Wout Mertens wrote:

> On Feb 12, 2009, at 3:13 PM, Kenneth Kalmer wrote:
>
>> Normal filesystem-based storage of mail has other issues:
>>
>> * Messages often smaller than ethernet jumbo frames, so limited  
>> throughput
>> (couch can overcome this by bundling messages in a single response)
>
> wow that's really specialized. Is that really an issue?
>
>> * Mostly limited by disk IO and clever tricks around solid state  
>> drive usage
>> or stripping excessively fast disks
>
> Hmmm makes me wonder - CouchDB doesn't change the position of  
> received documents on disk, so retrieving a view could mean many  
> disk accesses.
>
> Is that correct? Is that something all databases suffer from? I  
> suppose the only answer for that is caching?

The file and b-tree structure is actually made to require a minimal
number of disk head seeks (which is what makes disks slow) and
in a typical usage scenario, upper nodes of the btree are cached
in the filesystem buffer cache too, so there's another speedup.

Cheers
Jan
--

Re: [user] Couch as a mail store?

[Wout Mertens <wm...@cisco.com>]

On Feb 12, 2009, at 3:13 PM, Kenneth Kalmer wrote:

> Normal filesystem-based storage of mail has other issues:
>
> * Messages often smaller than ethernet jumbo frames, so limited  
> throughput
> (couch can overcome this by bundling messages in a single response)

wow that's really specialized. Is that really an issue?

> * Mostly limited by disk IO and clever tricks around solid state  
> drive usage
> or stripping excessively fast disks

Hmmm makes me wonder - CouchDB doesn't change the position of received  
documents on disk, so retrieving a view could mean many disk accesses.

Is that correct? Is that something all databases suffer from? I  
suppose the only answer for that is caching?

Wout.

Re: Couch as a mail store?

[Kenneth Kalmer <ke...@gmail.com>]

On Thu, Feb 12, 2009 at 7:24 PM, Jan Lehnardt <ja...@apache.org> wrote:

>
> On 12 Feb 2009, at 17:19, Brian Candler wrote:
>
>  Of course, SMTP clients don't mind a small delay before they get their 250
>> OK at the end of the message; you can therefore write a number of batches
>> and do an fsync() every second or so, as long as you remember not to send
>> the acknowledgement back to each client until *after* the fsync has
>> completed.
>>
>
> CouchDB does the fsync()-a-second currently.
>

Not that it matters though: Postfix (the only MTA I have experience with,
and years at it) spools the mail first into one of several queues. Once in
the queue it send the 250 back and the client is freed. The same should
happen before hitting couch, at least in my head. This also allows for
temporary failures between couch and the MTA without affecting the client.

But absolutely guaranteeing delivery is impossible, you can aim for 5 9's,
or 7 9's even, but all 100% won't happen...


 (2) Couch won't let you write a document to disk in chunks; if it doesn't
> get an up-front Content-Length: header then it will buffer the whole thing
> in RAM.
>
> So if you receive very large E-mail messages, you may wish to buffer them
> locally (e.g. in a tempfile on another disk) before sending them as a
> single
> document to Couch.
>
> Note that you sometimes get an indication of the message size in a SMTP
> transaction, but it's not guaranteed to be accurate; so you won't know the
> true size until you've read it in.
>

Sucky thing with email...

A patch to allow for streaming unknown-length attachments into CouchDB
> is in the works.
>

If you have a store-and-forward MTA you could live without this one, but it
sounds like a handy addition.

 (3) As you say, messages are stored and deleted frequently. You may end up
> having to compact your message store frequently, which means basically
> reading the whole store from start to end and rewriting it to a new file.
> This has to be done when write load isn't too high, to ensure that it will
> complete.
>

CouchDB's roadmap includes an item about pausing writes to allow compaction
> to catch up on high write loads.
>

Also nice, but again a store-and-forward MTA would help alleviate this
issue. It would be even better if the MTA could be made aware of the
compaction process and slow down ingestion.

Thanks for all the feedback so far, sounds like I've hit a common nail right
on the head here.


-- 
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za

Re: Couch as a mail store?

[Chris Anderson <jc...@apache.org>]

On Thu, Feb 12, 2009 at 10:01 PM, Brian Candler <B....@pobox.com> wrote:
> But I don't think this is currently exposed to the client. You'd need a
> "wait until next fsync has completed" request, to let you know that it's
> safe to send back a 250 OK to the client.
>

I'm working on a project that plans to use replication and the update
notification process to let clients know when a save is safely written
to 2 nodes (ignoring fsync issues...)

Write doc X to node A.
Always be replicating to B.
When doc X's new rev shows up on B, then you can tell the client it's
been saved.

-- 
Chris Anderson
http://jchris.mfdz.com

Re: Couch as a mail store?

[Kenneth Kalmer <ke...@gmail.com>]

On Fri, Feb 13, 2009 at 8:01 AM, Brian Candler <B....@pobox.com> wrote:

> > > Of course, SMTP clients don't mind a small delay before they get
> > > their 250
> > > OK at the end of the message; you can therefore write a number of
> > > batches
> > > and do an fsync() every second or so, as long as you remember not to
> > > send
> > > the acknowledgement back to each client until *after* the fsync has
> > > completed.
> >
> > CouchDB does the fsync()-a-second currently.
>
> But I don't think this is currently exposed to the client. You'd need a
> "wait until next fsync has completed" request, to let you know that it's
> safe to send back a 250 OK to the client.
>
> A couple of other thoughts:
>
> (4) E-mail systems create and delete millions of documents per day. Is it
> true that couchdb keeps a small amount of data around indefinitely for
> every
> deleted document, for replication purposes? If so this would keep growing.
> (And whilst I've seen discussion about pruning old _revs, I've not seen
> discussion of pruning deleted documents)


Very good point. Compaction will remove the document data, but not the rev's
(unless I'm mistaken). I'm putting my neck on a block here, just for
comparison, but MySQL's log replay-style replication avoids this issue. I
also want to state that I know it can't be compared reasonably because
couch's replication addresses a lot of other issues. Nevermind, I didn't
write the last sentence...


> (5) I believe the IMAP protocol makes some RDBMS-type demands on a
> mailstore, in particular the allocation of contiguous sequence numbers to
> each message. Some careful thought would be needed on how to do this, as it
> may make full replication difficult - or at least increase the likelihood
> of
> replication conflicts, so would benefit from the planned feature of being
> able to specify custom conflict-resolution logic.


Good point, but how do IMAP servers currently handle this? I don't think
it's the MTA's job to 'tag' the mails specially. Only thing I can think of
is the IMAP servers load the mails ordered by date and manage their own meta
data for things like 'read'. I have to admit I haven't really explored the
innerworkings of the IMAP protocol... I guess one could have extra
attributes in the docs for the IMAP requirements, but those should be
manipulated by the IMAP server directly. From my little work with Ruby's
Net::IMAP library I've seen that the mails have unique id's, which can
probably be the _id attribute?

As for the sequence, doesn't POP clients also request the number of messages
in the store and then pop them one by one (RETR N)? This would effectively
be the same problem for both, and having the results ordered by date and
then POP/IMAP handle the rest of the logic seems quite couch-ish.


> (6) Another planned/not-yet-complete feature which would be very useful for
> a mail store is the document-level access control logic, which could for
> example be used for IMAP ACLs and shared mailboxes.
>
>
ATM I'm mostly fine with this logic staying in the application. Even with my
current MySQL projects do we handle autharization in the applications
itself. But I understand if you want to expose your couch instance directly
to the world these things become an issue.

Ciao


-- 
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za

Re: Couch as a mail store?

[Jan Lehnardt <ja...@apache.org>]

Hi Brian,

On 13 Feb 2009, at 07:01, Brian Candler wrote:

> (4) E-mail systems create and delete millions of documents per day.  
> Is it
> true that couchdb keeps a small amount of data around indefinitely  
> for every
> deleted document, for replication purposes? If so this would keep  
> growing.
> (And whilst I've seen discussion about pruning old _revs, I've not  
> seen
> discussion of pruning deleted documents)

CouchDB has got purge. From couch_tests.js:

     var xhr = CouchDB.request("POST", "/test_suite_db/_purge", {
       body: JSON.stringify({"1":[doc1._rev], "2":[doc2._rev]}),
     });


Cheers
Jan
--
PS: Good discussion, keep it up!

Re: Couch as a mail store?

[Brian Candler <B....@pobox.com>]

> > Of course, SMTP clients don't mind a small delay before they get  
> > their 250
> > OK at the end of the message; you can therefore write a number of  
> > batches
> > and do an fsync() every second or so, as long as you remember not to  
> > send
> > the acknowledgement back to each client until *after* the fsync has
> > completed.
> 
> CouchDB does the fsync()-a-second currently.

But I don't think this is currently exposed to the client. You'd need a
"wait until next fsync has completed" request, to let you know that it's
safe to send back a 250 OK to the client.

A couple of other thoughts:

(4) E-mail systems create and delete millions of documents per day. Is it
true that couchdb keeps a small amount of data around indefinitely for every
deleted document, for replication purposes? If so this would keep growing.
(And whilst I've seen discussion about pruning old _revs, I've not seen
discussion of pruning deleted documents)

(5) I believe the IMAP protocol makes some RDBMS-type demands on a
mailstore, in particular the allocation of contiguous sequence numbers to
each message. Some careful thought would be needed on how to do this, as it
may make full replication difficult - or at least increase the likelihood of
replication conflicts, so would benefit from the planned feature of being
able to specify custom conflict-resolution logic.

(6) Another planned/not-yet-complete feature which would be very useful for
a mail store is the document-level access control logic, which could for
example be used for IMAP ACLs and shared mailboxes.

Regards,

Brian.

Re: Couch as a mail store?

[Jan Lehnardt <ja...@apache.org>]

On 12 Feb 2009, at 17:19, Brian Candler wrote:

> Of course, SMTP clients don't mind a small delay before they get  
> their 250
> OK at the end of the message; you can therefore write a number of  
> batches
> and do an fsync() every second or so, as long as you remember not to  
> send
> the acknowledgement back to each client until *after* the fsync has
> completed.

CouchDB does the fsync()-a-second currently.


> (2) Couch won't let you write a document to disk in chunks; if it  
> doesn't
> get an up-front Content-Length: header then it will buffer the whole  
> thing
> in RAM.
>
> So if you receive very large E-mail messages, you may wish to buffer  
> them
> locally (e.g. in a tempfile on another disk) before sending them as  
> a single
> document to Couch.
>
> Note that you sometimes get an indication of the message size in a  
> SMTP
> transaction, but it's not guaranteed to be accurate; so you won't  
> know the
> true size until you've read it in.


A patch to allow for streaming unknown-length attachments into CouchDB
is in the works.


> (3) As you say, messages are stored and deleted frequently. You may  
> end up
> having to compact your message store frequently, which means basically
> reading the whole store from start to end and rewriting it to a new  
> file.
> This has to be done when write load isn't too high, to ensure that  
> it will
> complete.

CouchDB's roadmap includes an item about pausing writes to allow  
compaction
to catch up on high write loads.

Cheers
Jan
--

Re: Couch as a mail store?

[Brian Candler <B....@pobox.com>]

On Thu, Feb 12, 2009 at 04:13:45PM +0200, Kenneth Kalmer wrote:
> How would couch fair as a backend for a mail delivery system (in concept)?
> Considering you need high availability and very fast IO. Documents (email
> messages) will be created and deleted very often, some almost
> instantaneously.

I was thinking about this too, and I think it would perform well. You could
for example store headers and IMAP flags in the JSON, and the body as an
attachment (or multiple attachments already MIME-broken down)

I can see a few things which need working through.

(1) One of the key performance issues with real-world SMTP servers is the
need to fsync() the file to disk before sending back an acknowledgement, in
order to guarantee delivery.

I asked about this recently, and it seems that Couch takes an optimistic
view: it writes the file to the OS but doesn't fsync unless you explicitly
ask it. There is a special HTTP header you can provide for this. If you
don't, then your database won't be corrupted if the plug is pulled, but it
may be missing data.

Of course, SMTP clients don't mind a small delay before they get their 250
OK at the end of the message; you can therefore write a number of batches
and do an fsync() every second or so, as long as you remember not to send
the acknowledgement back to each client until *after* the fsync has
completed.

(2) Couch won't let you write a document to disk in chunks; if it doesn't
get an up-front Content-Length: header then it will buffer the whole thing
in RAM.

So if you receive very large E-mail messages, you may wish to buffer them
locally (e.g. in a tempfile on another disk) before sending them as a single
document to Couch.

Note that you sometimes get an indication of the message size in a SMTP
transaction, but it's not guaranteed to be accurate; so you won't know the
true size until you've read it in.

(3) As you say, messages are stored and deleted frequently. You may end up
having to compact your message store frequently, which means basically
reading the whole store from start to end and rewriting it to a new file.
This has to be done when write load isn't too high, to ensure that it will
complete.

Regards,

Brian.