You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Dave Ray <da...@gmail.com> on 2009/12/04 02:48:55 UTC
Documents with many attachments
Hi there,
Is there any reason in terms of performance, stability, or good taste,
that I should avoid adding 100s or 1000s of attachments to a document?
I have about 3000 pngs on a document and things seem to work fine
(aside from Futon being a little unusable when I view the doc), but if
this is going to cause me trouble later I can split it up into
multiple docs with single attachments without much more work. Just
wondering.
Thanks!
Dave
Re: Documents with many attachments
Posted by Dave Ray <da...@gmail.com>.
Thanks everybody for all the feedback. Having so many PNGs is, in
itself, a flaw in the current system so there's a good chance this may
just go away as we move forward. It's good to know about these
potential problems ahead of time though.
Dave
On Fri, Dec 4, 2009 at 4:34 AM, Paul Davis <pa...@gmail.com> wrote:
>>> One of the CouchDB config files defines the maximum document size as 4GB.
>>> Does that include attachments?
>>>
>>
>> No, but it may include attachment stubs (although I'm guessing it doesn't).
>>
>
> That's the mochiweb request body size number isn't it? So it would be
> "maximum size of any request to couchdb."
>
> Paul Davis
>
Re: Documents with many attachments
Posted by Paul Davis <pa...@gmail.com>.
>> One of the CouchDB config files defines the maximum document size as 4GB.
>> Does that include attachments?
>>
>
> No, but it may include attachment stubs (although I'm guessing it doesn't).
>
That's the mochiweb request body size number isn't it? So it would be
"maximum size of any request to couchdb."
Paul Davis
Re: Documents with many attachments
Posted by Chris Anderson <jc...@apache.org>.
On Thu, Dec 3, 2009 at 9:14 PM, Roger Binns <ro...@rogerbinns.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Chris Anderson wrote:
>> At a certain point, you could have too many, and that would start to
>> suck. Since they are all in one doc the metadata for the attachments
>> has to fit in memory.
>
> One of the CouchDB config files defines the maximum document size as 4GB.
> Does that include attachments?
>
No, but it may include attachment stubs (although I'm guessing it doesn't).
> Roger
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAksYmqgACgkQmOOfHg372QRWEwCfa1xqQGGzTR7MnDSEtP+1Ej9Q
> 9ZQAn18b895Xa4pRPucZ57V+CS9GWnDt
> =rrwq
> -----END PGP SIGNATURE-----
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Documents with many attachments
Posted by Roger Binns <ro...@rogerbinns.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chris Anderson wrote:
> At a certain point, you could have too many, and that would start to
> suck. Since they are all in one doc the metadata for the attachments
> has to fit in memory.
One of the CouchDB config files defines the maximum document size as 4GB.
Does that include attachments?
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAksYmqgACgkQmOOfHg372QRWEwCfa1xqQGGzTR7MnDSEtP+1Ej9Q
9ZQAn18b895Xa4pRPucZ57V+CS9GWnDt
=rrwq
-----END PGP SIGNATURE-----
Re: Documents with many attachments
Posted by Adam Kocoloski <ko...@apache.org>.
On Dec 3, 2009, at 10:57 PM, Paul Davis wrote:
> On Thu, Dec 3, 2009 at 10:43 PM, Chris Anderson <jc...@apache.org> wrote:
>> On Thu, Dec 3, 2009 at 5:48 PM, Dave Ray <da...@gmail.com> wrote:
>>> Hi there,
>>>
>>> Is there any reason in terms of performance, stability, or good taste,
>>> that I should avoid adding 100s or 1000s of attachments to a document?
>>> I have about 3000 pngs on a document and things seem to work fine
>>> (aside from Futon being a little unusable when I view the doc), but if
>>> this is going to cause me trouble later I can split it up into
>>> multiple docs with single attachments without much more work. Just
>>> wondering.
>>
>> At a certain point, you could have too many, and that would start to
>> suck. Since they are all in one doc the metadata for the attachments
>> has to fit in memory.
>>
>> Probably the best metric you can have is the size of the JSON stubs
>> you're schlepping around on the doc. With 100k attachments the JSON
>> doc would be too big to deal with. Maybe you can handle all the JSON
>> for 10k attachments -- a doc with one thousand attachments is not that
>> many kb of JSON.
>>
>
> There's also one of the replication directions. I don't remember
> which, but one of them doesn't like having lots or large attachments.
> This might be changing as we got to a multipart mime format, but for
> 0.10.x or before the issue exists.
>
> Paul Davis
Push replication (local source, remote target) is the one that will fail badly with this many attachments in 0.10.x or lower. Should be better in the next release, as Paul points out. Best,
Adam
Re: Documents with many attachments
Posted by Paul Davis <pa...@gmail.com>.
On Thu, Dec 3, 2009 at 10:43 PM, Chris Anderson <jc...@apache.org> wrote:
> On Thu, Dec 3, 2009 at 5:48 PM, Dave Ray <da...@gmail.com> wrote:
>> Hi there,
>>
>> Is there any reason in terms of performance, stability, or good taste,
>> that I should avoid adding 100s or 1000s of attachments to a document?
>> I have about 3000 pngs on a document and things seem to work fine
>> (aside from Futon being a little unusable when I view the doc), but if
>> this is going to cause me trouble later I can split it up into
>> multiple docs with single attachments without much more work. Just
>> wondering.
>
> At a certain point, you could have too many, and that would start to
> suck. Since they are all in one doc the metadata for the attachments
> has to fit in memory.
>
> Probably the best metric you can have is the size of the JSON stubs
> you're schlepping around on the doc. With 100k attachments the JSON
> doc would be too big to deal with. Maybe you can handle all the JSON
> for 10k attachments -- a doc with one thousand attachments is not that
> many kb of JSON.
>
There's also one of the replication directions. I don't remember
which, but one of them doesn't like having lots or large attachments.
This might be changing as we got to a multipart mime format, but for
0.10.x or before the issue exists.
Paul Davis
Re: Documents with many attachments
Posted by Chris Anderson <jc...@apache.org>.
On Thu, Dec 3, 2009 at 5:48 PM, Dave Ray <da...@gmail.com> wrote:
> Hi there,
>
> Is there any reason in terms of performance, stability, or good taste,
> that I should avoid adding 100s or 1000s of attachments to a document?
> I have about 3000 pngs on a document and things seem to work fine
> (aside from Futon being a little unusable when I view the doc), but if
> this is going to cause me trouble later I can split it up into
> multiple docs with single attachments without much more work. Just
> wondering.
At a certain point, you could have too many, and that would start to
suck. Since they are all in one doc the metadata for the attachments
has to fit in memory.
Probably the best metric you can have is the size of the JSON stubs
you're schlepping around on the doc. With 100k attachments the JSON
doc would be too big to deal with. Maybe you can handle all the JSON
for 10k attachments -- a doc with one thousand attachments is not that
many kb of JSON.
>
> Thanks!
>
> Dave
>
--
Chris Anderson
http://jchrisa.net
http://couch.io