You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Mark Deibert <ma...@gmail.com> on 2013/11/09 16:15:10 UTC
Storage limitations?
I have two storage questions. How big can any one field in one doc be?
Also, is there any limit on the size of a CouchDB? Meaning, if I have 1TB
of free disk space, will the db grow to 1TB in size?
Re: Storage limitations?
Posted by Mark Deibert <ma...@gmail.com>.
So Couch will not handle lots (I hope many thousands eventually) of 3MB
photo attachments? This really stinks. So now I need to bring back a PHP
server script to support my Couchapp?
On Sun, Nov 10, 2013 at 4:10 PM, Mark Deibert <ma...@gmail.com>wrote:
> Each comment is normally a very small amount of information, and id,
> userName, a date and a small text field. Probably no more than a short
> sentence on average. Why do I need to go through the trouble of creating
> new comment docs for each? This totally complicates the comment read code
> (there will 1000 reads to every 1 write, at least) both in the UI and the
> db for no reason.
>
>
> On Sun, Nov 10, 2013 at 1:34 PM, Jens Alfke <je...@couchbase.com> wrote:
>
>>
>> On Nov 9, 2013, at 11:47 AM, Robert Newson <rnewson@apache.org<mailto:
>> rnewson@apache.org>> wrote:
>>
>> attachments are different to documents. They're stored as a series of
>> binary chunks and so they can be streamed in and out, you can go large
>> with attachments.
>>
>> But on the other hand, all attachments will get copied during a database
>> compaction, so they slow down the process and require more free disk space.
>> If you have many gigabytes of attachments, you might consider storing them
>> externally and putting URL links in the documents.
>>
>> As for comments, just add new documents for each comment and use a
>> view (https://wiki.apache.org/couchdb/HTTP_view_API,
>> https://wiki.apache.org/couchdb/View_collation) to bring the article
>> and comment thread together. No need to update a document that way.
>>
>> Yup. The guide <guide.couchdb.org<http://guide.couchdb.org>> has a
>> chapter-long example of a blog application that shows how to do comments
>> this way.
>>
>> —Jens
>>
>
>
Re: Storage limitations?
Posted by Stanley Iriele <si...@gmail.com>.
Its technically faster to not include docs and to just emit the doc but it
takes far more space because you are pretty much copying it....I would say
just grab what you need and emit that as the value if its now that much.
Re: Storage limitations?
Posted by Simon Metson <si...@cloudant.com>.
I think if you can happily fit it all into CouchDB (and from the numbers you’ve mentioned so far I’m confident you can) I’d go with that - simplest solution. If your 1000’s become 1000000’s then you can bite the bullet and switch to metadata in CouchDB, image data in something else.
On Monday, 11 November 2013 at 15:16, Mark Deibert wrote:
> @Simon: I understand what your saying about "source of truth". What is
> master record, the files or the Couch records if the upload succeeds and
> the Couch record write fails or vise-versa. Not sure what the answer is
> here either.
>
>
> On Mon, Nov 11, 2013 at 10:00 AM, Mark Deibert <mark.deibert@gmail.com (mailto:mark.deibert@gmail.com)>wrote:
>
> > @Simon: Thanks for the advice. As I type, I don't anticipate any
> > aggregation on the Photo docs, but I should think about that some more.
> >
> >
> > On Mon, Nov 11, 2013 at 9:56 AM, Simon Metson <simon@cloudant.com (mailto:simon@cloudant.com)> wrote:
> >
> > > Hey,
> > > I’d go with what you’re doing (1:1 doc:photo).
> > >
> > > 2 seems a bit weird - do you mean host the app in a different server to
> > > the data or have different databases for different types of data (that
> > > might work, if you never want to query across types of data)? 3 might work
> > > so long as you never want to aggregate across the categories. 4 is a
> > > reasonable approach for very very large attachments, but you can get into
> > > consistency issues - what’s the source of truth the database or the
> > > filesystem?
> > > Cheers
> > > Simon
> > >
> > >
> > > On Monday, 11 November 2013 at 14:41, Mark Deibert wrote:
> > >
> > > > A followup on the "1000's of images" question. I could approach this a
> > > > couple ways. Currently each image is attached to it's own Photo doc.
> > >
> > >
> > > Which
> > > > I've read is better for replication than one attachment with many
> > > > attachments. So that's fine, but will Couch have any issue managing
> > >
> > >
> > > several
> > > > thousand of these Photo docs, each with a 3MB'ish image attachment? If
> > >
> > >
> > > you
> > > > were building this Couchapp, would you...
> > > >
> > > > 1) Keep the photos as described above in one CouchDB
> > > > 2) Move the Photo docs with attachments out into a separate CouchDB
> > > > 3) Do 2, but break Photos into multiple categorized CouchDBs
> > > > 4) Upload the images to the filesystem, just store the link in Couch
> > > >
> > > > I want to build this Couchapp in such a way as to not make life
> > > miserable
> > > > for CouchDB :-D
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, Nov 10, 2013 at 6:33 PM, Dave Cottlehuber <dch@jsonified.com (mailto:dch@jsonified.com)(mailto:
> > > dch@jsonified.com (mailto:dch@jsonified.com))> wrote:
> > > >
> > > > > On 10 November 2013 23:14, Mark Deibert <mark.deibert@gmail.com (mailto:mark.deibert@gmail.com)(mailto:
> > > mark.deibert@gmail.com (mailto:mark.deibert@gmail.com))> wrote:
> > > > > > I read an article somewhere that using include_docs is "hard" on
> > > > >
> > > >
> > >
> > >
> > > memory
> > > > >
> > > > >
> > > > > or
> > > > > > disk or in some way taxes Couch and therefore you should just emit
> > > > >
> > > >
> > >
> > >
> > > the
> > > > >
> > > > >
> > > > > doc.
> > > > > > Is this true?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Like most general statements it has some truth and some lies in it :-)
> > > > >
> > > > > views and docs are stored in separate .couch btree files on disk.
> > > > >
> > > > > emit(key, doc) puts a full copy of the doc (that's already in the doc
> > > > > .couch b~tree) into the view b~tree.
> > > > >
> > > > > advantage - no need to hop over to the doc .couch file to retrieve the
> > > > > document.
> > > > > disadvantage - you now have 2 copies of the doc in separate files,
> > > >
> > >
> > >
> > > wasted
> > > > > space.
> > > > >
> > > > > If you do things right, and your app fits this model, the generated
> > > > > etags from views and docs can be cached in nginx or similar, and
> > > > > repeated queries don't need to hit your couch.
> > > > >
> > > > > So yes, include_docs means extra reads, but like most of these things
> > > > > you should benchmark your situation, under a realistic load, not just
> > > > > pumping 1000 single-doc reads at it.
> > > > >
> > > > > A+
> > > > > Dave
> > > >
> > >
> >
>
Re: Storage limitations?
Posted by Mark Deibert <ma...@gmail.com>.
@Simon: I understand what your saying about "source of truth". What is
master record, the files or the Couch records if the upload succeeds and
the Couch record write fails or vise-versa. Not sure what the answer is
here either.
On Mon, Nov 11, 2013 at 10:00 AM, Mark Deibert <ma...@gmail.com>wrote:
> @Simon: Thanks for the advice. As I type, I don't anticipate any
> aggregation on the Photo docs, but I should think about that some more.
>
>
> On Mon, Nov 11, 2013 at 9:56 AM, Simon Metson <si...@cloudant.com> wrote:
>
>> Hey,
>> I’d go with what you’re doing (1:1 doc:photo).
>>
>> 2 seems a bit weird - do you mean host the app in a different server to
>> the data or have different databases for different types of data (that
>> might work, if you never want to query across types of data)? 3 might work
>> so long as you never want to aggregate across the categories. 4 is a
>> reasonable approach for very very large attachments, but you can get into
>> consistency issues - what’s the source of truth the database or the
>> filesystem?
>> Cheers
>> Simon
>>
>>
>> On Monday, 11 November 2013 at 14:41, Mark Deibert wrote:
>>
>> > A followup on the "1000's of images" question. I could approach this a
>> > couple ways. Currently each image is attached to it's own Photo doc.
>> Which
>> > I've read is better for replication than one attachment with many
>> > attachments. So that's fine, but will Couch have any issue managing
>> several
>> > thousand of these Photo docs, each with a 3MB'ish image attachment? If
>> you
>> > were building this Couchapp, would you...
>> >
>> > 1) Keep the photos as described above in one CouchDB
>> > 2) Move the Photo docs with attachments out into a separate CouchDB
>> > 3) Do 2, but break Photos into multiple categorized CouchDBs
>> > 4) Upload the images to the filesystem, just store the link in Couch
>> >
>> > I want to build this Couchapp in such a way as to not make life
>> miserable
>> > for CouchDB :-D
>> >
>> >
>> >
>> >
>> > On Sun, Nov 10, 2013 at 6:33 PM, Dave Cottlehuber <dch@jsonified.com(mailto:
>> dch@jsonified.com)> wrote:
>> >
>> > > On 10 November 2013 23:14, Mark Deibert <mark.deibert@gmail.com(mailto:
>> mark.deibert@gmail.com)> wrote:
>> > > > I read an article somewhere that using include_docs is "hard" on
>> memory
>> > >
>> > >
>> > > or
>> > > > disk or in some way taxes Couch and therefore you should just emit
>> the
>> > >
>> > >
>> > > doc.
>> > > > Is this true?
>> > >
>> > >
>> > >
>> > > Like most general statements it has some truth and some lies in it :-)
>> > >
>> > > views and docs are stored in separate .couch btree files on disk.
>> > >
>> > > emit(key, doc) puts a full copy of the doc (that's already in the doc
>> > > .couch b~tree) into the view b~tree.
>> > >
>> > > advantage - no need to hop over to the doc .couch file to retrieve the
>> > > document.
>> > > disadvantage - you now have 2 copies of the doc in separate files,
>> wasted
>> > > space.
>> > >
>> > > If you do things right, and your app fits this model, the generated
>> > > etags from views and docs can be cached in nginx or similar, and
>> > > repeated queries don't need to hit your couch.
>> > >
>> > > So yes, include_docs means extra reads, but like most of these things
>> > > you should benchmark your situation, under a realistic load, not just
>> > > pumping 1000 single-doc reads at it.
>> > >
>> > > A+
>> > > Dave
>> >
>>
>>
>>
>>
>
Re: Storage limitations?
Posted by Mark Deibert <ma...@gmail.com>.
@Simon: Thanks for the advice. As I type, I don't anticipate any
aggregation on the Photo docs, but I should think about that some more.
On Mon, Nov 11, 2013 at 9:56 AM, Simon Metson <si...@cloudant.com> wrote:
> Hey,
> I’d go with what you’re doing (1:1 doc:photo).
>
> 2 seems a bit weird - do you mean host the app in a different server to
> the data or have different databases for different types of data (that
> might work, if you never want to query across types of data)? 3 might work
> so long as you never want to aggregate across the categories. 4 is a
> reasonable approach for very very large attachments, but you can get into
> consistency issues - what’s the source of truth the database or the
> filesystem?
> Cheers
> Simon
>
>
> On Monday, 11 November 2013 at 14:41, Mark Deibert wrote:
>
> > A followup on the "1000's of images" question. I could approach this a
> > couple ways. Currently each image is attached to it's own Photo doc.
> Which
> > I've read is better for replication than one attachment with many
> > attachments. So that's fine, but will Couch have any issue managing
> several
> > thousand of these Photo docs, each with a 3MB'ish image attachment? If
> you
> > were building this Couchapp, would you...
> >
> > 1) Keep the photos as described above in one CouchDB
> > 2) Move the Photo docs with attachments out into a separate CouchDB
> > 3) Do 2, but break Photos into multiple categorized CouchDBs
> > 4) Upload the images to the filesystem, just store the link in Couch
> >
> > I want to build this Couchapp in such a way as to not make life miserable
> > for CouchDB :-D
> >
> >
> >
> >
> > On Sun, Nov 10, 2013 at 6:33 PM, Dave Cottlehuber <dch@jsonified.com(mailto:
> dch@jsonified.com)> wrote:
> >
> > > On 10 November 2013 23:14, Mark Deibert <mark.deibert@gmail.com(mailto:
> mark.deibert@gmail.com)> wrote:
> > > > I read an article somewhere that using include_docs is "hard" on
> memory
> > >
> > >
> > > or
> > > > disk or in some way taxes Couch and therefore you should just emit
> the
> > >
> > >
> > > doc.
> > > > Is this true?
> > >
> > >
> > >
> > > Like most general statements it has some truth and some lies in it :-)
> > >
> > > views and docs are stored in separate .couch btree files on disk.
> > >
> > > emit(key, doc) puts a full copy of the doc (that's already in the doc
> > > .couch b~tree) into the view b~tree.
> > >
> > > advantage - no need to hop over to the doc .couch file to retrieve the
> > > document.
> > > disadvantage - you now have 2 copies of the doc in separate files,
> wasted
> > > space.
> > >
> > > If you do things right, and your app fits this model, the generated
> > > etags from views and docs can be cached in nginx or similar, and
> > > repeated queries don't need to hit your couch.
> > >
> > > So yes, include_docs means extra reads, but like most of these things
> > > you should benchmark your situation, under a realistic load, not just
> > > pumping 1000 single-doc reads at it.
> > >
> > > A+
> > > Dave
> >
>
>
>
>
Re: Storage limitations?
Posted by Simon Metson <si...@cloudant.com>.
Hey,
I’d go with what you’re doing (1:1 doc:photo).
2 seems a bit weird - do you mean host the app in a different server to the data or have different databases for different types of data (that might work, if you never want to query across types of data)? 3 might work so long as you never want to aggregate across the categories. 4 is a reasonable approach for very very large attachments, but you can get into consistency issues - what’s the source of truth the database or the filesystem?
Cheers
Simon
On Monday, 11 November 2013 at 14:41, Mark Deibert wrote:
> A followup on the "1000's of images" question. I could approach this a
> couple ways. Currently each image is attached to it's own Photo doc. Which
> I've read is better for replication than one attachment with many
> attachments. So that's fine, but will Couch have any issue managing several
> thousand of these Photo docs, each with a 3MB'ish image attachment? If you
> were building this Couchapp, would you...
>
> 1) Keep the photos as described above in one CouchDB
> 2) Move the Photo docs with attachments out into a separate CouchDB
> 3) Do 2, but break Photos into multiple categorized CouchDBs
> 4) Upload the images to the filesystem, just store the link in Couch
>
> I want to build this Couchapp in such a way as to not make life miserable
> for CouchDB :-D
>
>
>
>
> On Sun, Nov 10, 2013 at 6:33 PM, Dave Cottlehuber <dch@jsonified.com (mailto:dch@jsonified.com)> wrote:
>
> > On 10 November 2013 23:14, Mark Deibert <mark.deibert@gmail.com (mailto:mark.deibert@gmail.com)> wrote:
> > > I read an article somewhere that using include_docs is "hard" on memory
> >
> >
> > or
> > > disk or in some way taxes Couch and therefore you should just emit the
> >
> >
> > doc.
> > > Is this true?
> >
> >
> >
> > Like most general statements it has some truth and some lies in it :-)
> >
> > views and docs are stored in separate .couch btree files on disk.
> >
> > emit(key, doc) puts a full copy of the doc (that's already in the doc
> > .couch b~tree) into the view b~tree.
> >
> > advantage - no need to hop over to the doc .couch file to retrieve the
> > document.
> > disadvantage - you now have 2 copies of the doc in separate files, wasted
> > space.
> >
> > If you do things right, and your app fits this model, the generated
> > etags from views and docs can be cached in nginx or similar, and
> > repeated queries don't need to hit your couch.
> >
> > So yes, include_docs means extra reads, but like most of these things
> > you should benchmark your situation, under a realistic load, not just
> > pumping 1000 single-doc reads at it.
> >
> > A+
> > Dave
>
Re: Storage limitations?
Posted by Mark Deibert <ma...@gmail.com>.
A followup on the "1000's of images" question. I could approach this a
couple ways. Currently each image is attached to it's own Photo doc. Which
I've read is better for replication than one attachment with many
attachments. So that's fine, but will Couch have any issue managing several
thousand of these Photo docs, each with a 3MB'ish image attachment? If you
were building this Couchapp, would you...
1) Keep the photos as described above in one CouchDB
2) Move the Photo docs with attachments out into a separate CouchDB
3) Do 2, but break Photos into multiple categorized CouchDBs
4) Upload the images to the filesystem, just store the link in Couch
I want to build this Couchapp in such a way as to not make life miserable
for CouchDB :-D
On Sun, Nov 10, 2013 at 6:33 PM, Dave Cottlehuber <dc...@jsonified.com> wrote:
> On 10 November 2013 23:14, Mark Deibert <ma...@gmail.com> wrote:
> > I read an article somewhere that using include_docs is "hard" on memory
> or
> > disk or in some way taxes Couch and therefore you should just emit the
> doc.
> > Is this true?
>
> Like most general statements it has some truth and some lies in it :-)
>
> views and docs are stored in separate .couch btree files on disk.
>
> emit(key, doc) puts a full copy of the doc (that's already in the doc
> .couch b~tree) into the view b~tree.
>
> advantage - no need to hop over to the doc .couch file to retrieve the
> document.
> disadvantage - you now have 2 copies of the doc in separate files, wasted
> space.
>
> If you do things right, and your app fits this model, the generated
> etags from views and docs can be cached in nginx or similar, and
> repeated queries don't need to hit your couch.
>
> So yes, include_docs means extra reads, but like most of these things
> you should benchmark your situation, under a realistic load, not just
> pumping 1000 single-doc reads at it.
>
> A+
> Dave
>
Re: Storage limitations?
Posted by Dave Cottlehuber <dc...@jsonified.com>.
On 10 November 2013 23:14, Mark Deibert <ma...@gmail.com> wrote:
> I read an article somewhere that using include_docs is "hard" on memory or
> disk or in some way taxes Couch and therefore you should just emit the doc.
> Is this true?
Like most general statements it has some truth and some lies in it :-)
views and docs are stored in separate .couch btree files on disk.
emit(key, doc) puts a full copy of the doc (that's already in the doc
.couch b~tree) into the view b~tree.
advantage - no need to hop over to the doc .couch file to retrieve the document.
disadvantage - you now have 2 copies of the doc in separate files, wasted space.
If you do things right, and your app fits this model, the generated
etags from views and docs can be cached in nginx or similar, and
repeated queries don't need to hit your couch.
So yes, include_docs means extra reads, but like most of these things
you should benchmark your situation, under a realistic load, not just
pumping 1000 single-doc reads at it.
A+
Dave
Re: Storage limitations?
Posted by Mark Deibert <ma...@gmail.com>.
I read an article somewhere that using include_docs is "hard" on memory or
disk or in some way taxes Couch and therefore you should just emit the doc.
Is this true?
On Sun, Nov 10, 2013 at 5:11 PM, Mark Deibert <ma...@gmail.com>wrote:
> BTW, my current code _is_ storing each comment as it's own doc, and I
> already _do have_ and am using the view that is being described here. I was
> just trying to figure out something more denormalized. However, Keith's
> comment about the conflicts might be nail. I didn't think about that.
>
>
> On Sun, Nov 10, 2013 at 5:00 PM, Mike Marino <mm...@gmail.com> wrote:
>
>> Hi Mark,
>>
>> On Sun, Nov 10, 2013 at 10:10 PM, Mark Deibert <ma...@gmail.com>
>> wrote:
>> > Each comment is normally a very small amount of information, and id,
>> > userName, a date and a small text field. Probably no more than a short
>> > sentence on average. Why do I need to go through the trouble of creating
>> > new comment docs for each? This totally complicates the comment read
>> code
>> > (there will 1000 reads to every 1 write, at least) both in the UI and
>> the
>> > db for no reason.
>>
>> I'm not sure why this should complicate the comment read code. You
>> can generate a view that emits as key the particular page/post to
>> which the comment refers. As value, you can emit either a subset of
>> data you need from the comment, or simply emit "null" and call the
>> view with include_docs to get all the documents for the particular
>> post. This would be a single http call.
>>
>> Cheers,
>> Mike
>>
>> >
>> >
>> > On Sun, Nov 10, 2013 at 1:34 PM, Jens Alfke <je...@couchbase.com> wrote:
>> >
>> >>
>> >> On Nov 9, 2013, at 11:47 AM, Robert Newson <rnewson@apache.org<mailto:
>> >> rnewson@apache.org>> wrote:
>> >>
>> >> attachments are different to documents. They're stored as a series of
>> >> binary chunks and so they can be streamed in and out, you can go large
>> >> with attachments.
>> >>
>> >> But on the other hand, all attachments will get copied during a
>> database
>> >> compaction, so they slow down the process and require more free disk
>> space.
>> >> If you have many gigabytes of attachments, you might consider storing
>> them
>> >> externally and putting URL links in the documents.
>> >>
>> >> As for comments, just add new documents for each comment and use a
>> >> view (https://wiki.apache.org/couchdb/HTTP_view_API,
>> >> https://wiki.apache.org/couchdb/View_collation) to bring the article
>> >> and comment thread together. No need to update a document that way.
>> >>
>> >> Yup. The guide <guide.couchdb.org<http://guide.couchdb.org>> has a
>> >> chapter-long example of a blog application that shows how to do
>> comments
>> >> this way.
>> >>
>> >> —Jens
>> >>
>>
>
>
Re: Storage limitations?
Posted by Mark Deibert <ma...@gmail.com>.
BTW, my current code _is_ storing each comment as it's own doc, and I
already _do have_ and am using the view that is being described here. I was
just trying to figure out something more denormalized. However, Keith's
comment about the conflicts might be nail. I didn't think about that.
On Sun, Nov 10, 2013 at 5:00 PM, Mike Marino <mm...@gmail.com> wrote:
> Hi Mark,
>
> On Sun, Nov 10, 2013 at 10:10 PM, Mark Deibert <ma...@gmail.com>
> wrote:
> > Each comment is normally a very small amount of information, and id,
> > userName, a date and a small text field. Probably no more than a short
> > sentence on average. Why do I need to go through the trouble of creating
> > new comment docs for each? This totally complicates the comment read code
> > (there will 1000 reads to every 1 write, at least) both in the UI and the
> > db for no reason.
>
> I'm not sure why this should complicate the comment read code. You
> can generate a view that emits as key the particular page/post to
> which the comment refers. As value, you can emit either a subset of
> data you need from the comment, or simply emit "null" and call the
> view with include_docs to get all the documents for the particular
> post. This would be a single http call.
>
> Cheers,
> Mike
>
> >
> >
> > On Sun, Nov 10, 2013 at 1:34 PM, Jens Alfke <je...@couchbase.com> wrote:
> >
> >>
> >> On Nov 9, 2013, at 11:47 AM, Robert Newson <rnewson@apache.org<mailto:
> >> rnewson@apache.org>> wrote:
> >>
> >> attachments are different to documents. They're stored as a series of
> >> binary chunks and so they can be streamed in and out, you can go large
> >> with attachments.
> >>
> >> But on the other hand, all attachments will get copied during a database
> >> compaction, so they slow down the process and require more free disk
> space.
> >> If you have many gigabytes of attachments, you might consider storing
> them
> >> externally and putting URL links in the documents.
> >>
> >> As for comments, just add new documents for each comment and use a
> >> view (https://wiki.apache.org/couchdb/HTTP_view_API,
> >> https://wiki.apache.org/couchdb/View_collation) to bring the article
> >> and comment thread together. No need to update a document that way.
> >>
> >> Yup. The guide <guide.couchdb.org<http://guide.couchdb.org>> has a
> >> chapter-long example of a blog application that shows how to do comments
> >> this way.
> >>
> >> —Jens
> >>
>
Re: Storage limitations?
Posted by Mike Marino <mm...@gmail.com>.
Hi Mark,
On Sun, Nov 10, 2013 at 10:10 PM, Mark Deibert <ma...@gmail.com> wrote:
> Each comment is normally a very small amount of information, and id,
> userName, a date and a small text field. Probably no more than a short
> sentence on average. Why do I need to go through the trouble of creating
> new comment docs for each? This totally complicates the comment read code
> (there will 1000 reads to every 1 write, at least) both in the UI and the
> db for no reason.
I'm not sure why this should complicate the comment read code. You
can generate a view that emits as key the particular page/post to
which the comment refers. As value, you can emit either a subset of
data you need from the comment, or simply emit "null" and call the
view with include_docs to get all the documents for the particular
post. This would be a single http call.
Cheers,
Mike
>
>
> On Sun, Nov 10, 2013 at 1:34 PM, Jens Alfke <je...@couchbase.com> wrote:
>
>>
>> On Nov 9, 2013, at 11:47 AM, Robert Newson <rnewson@apache.org<mailto:
>> rnewson@apache.org>> wrote:
>>
>> attachments are different to documents. They're stored as a series of
>> binary chunks and so they can be streamed in and out, you can go large
>> with attachments.
>>
>> But on the other hand, all attachments will get copied during a database
>> compaction, so they slow down the process and require more free disk space.
>> If you have many gigabytes of attachments, you might consider storing them
>> externally and putting URL links in the documents.
>>
>> As for comments, just add new documents for each comment and use a
>> view (https://wiki.apache.org/couchdb/HTTP_view_API,
>> https://wiki.apache.org/couchdb/View_collation) to bring the article
>> and comment thread together. No need to update a document that way.
>>
>> Yup. The guide <guide.couchdb.org<http://guide.couchdb.org>> has a
>> chapter-long example of a blog application that shows how to do comments
>> this way.
>>
>> —Jens
>>
Re: Storage limitations?
Posted by Mark Deibert <ma...@gmail.com>.
Each comment is normally a very small amount of information, and id,
userName, a date and a small text field. Probably no more than a short
sentence on average. Why do I need to go through the trouble of creating
new comment docs for each? This totally complicates the comment read code
(there will 1000 reads to every 1 write, at least) both in the UI and the
db for no reason.
On Sun, Nov 10, 2013 at 1:34 PM, Jens Alfke <je...@couchbase.com> wrote:
>
> On Nov 9, 2013, at 11:47 AM, Robert Newson <rnewson@apache.org<mailto:
> rnewson@apache.org>> wrote:
>
> attachments are different to documents. They're stored as a series of
> binary chunks and so they can be streamed in and out, you can go large
> with attachments.
>
> But on the other hand, all attachments will get copied during a database
> compaction, so they slow down the process and require more free disk space.
> If you have many gigabytes of attachments, you might consider storing them
> externally and putting URL links in the documents.
>
> As for comments, just add new documents for each comment and use a
> view (https://wiki.apache.org/couchdb/HTTP_view_API,
> https://wiki.apache.org/couchdb/View_collation) to bring the article
> and comment thread together. No need to update a document that way.
>
> Yup. The guide <guide.couchdb.org<http://guide.couchdb.org>> has a
> chapter-long example of a blog application that shows how to do comments
> this way.
>
> —Jens
>
Re: Storage limitations?
Posted by Jens Alfke <je...@couchbase.com>.
On Nov 9, 2013, at 11:47 AM, Robert Newson <rn...@apache.org>> wrote:
attachments are different to documents. They're stored as a series of
binary chunks and so they can be streamed in and out, you can go large
with attachments.
But on the other hand, all attachments will get copied during a database compaction, so they slow down the process and require more free disk space. If you have many gigabytes of attachments, you might consider storing them externally and putting URL links in the documents.
As for comments, just add new documents for each comment and use a
view (https://wiki.apache.org/couchdb/HTTP_view_API,
https://wiki.apache.org/couchdb/View_collation) to bring the article
and comment thread together. No need to update a document that way.
Yup. The guide <guide.couchdb.org<http://guide.couchdb.org>> has a chapter-long example of a blog application that shows how to do comments this way.
—Jens
Re: Storage limitations?
Posted by Robert Newson <rn...@apache.org>.
attachments are different to documents. They're stored as a series of
binary chunks and so they can be streamed in and out, you can go large
with attachments. The reason I say documents can't be too huge is that
couchdb has to hold them in memory to process them or even return
them.
As for comments, just add new documents for each comment and use a
view (https://wiki.apache.org/couchdb/HTTP_view_API,
https://wiki.apache.org/couchdb/View_collation) to bring the article
and comment thread together. No need to update a document that way.
B.
On 9 November 2013 19:31, Mark Hahn <ma...@reevuit.com> wrote:
>> sorting/displaying a comments thread with replies by appending the replies
> within their respective original comment field.
>
> Wouldn't it be more natural to use an array? And it would be more flexible
> for possible future changes. It doesn't affect the size of the field but
> there is no way a comment thread is going to get up to even a megabyte.
> The entire bible fits in a megabyte.
>
>
> On Sat, Nov 9, 2013 at 9:22 AM, Mark Deibert <ma...@gmail.com> wrote:
>
>> Ok I was assuming the db could be really big. I'm storing images so it
>> _will_ get pretty large.
>>
>> For the field size question: I had an idea to greatly simplify
>> sorting/displaying a comments thread with replies by appending the replies
>> within their respective original comment field. Since the order I want to
>> display the comments is the natural chronological order, a simple append
>> process seems to make sense. Haven't tried yet but seems logical.
>>
>>
>> On Sat, Nov 9, 2013 at 11:32 AM, Robert Newson <rn...@apache.org> wrote:
>>
>> > while the default configuration has max_document_size as 4Gb, it would
>> > be very unwise to make a document anywhere near that large. Database
>> > size is limited only by disk capacity but note that you'll need some
>> > free space for compacting the database.
>> >
>> > On 9 November 2013 15:28, first-leon@yandex.ru <fi...@yandex.ru>
>> > wrote:
>> > > Base size in no limit. But file system may be limit by file size.
>> > > Doc size limit is 4Gb. (one or two or more field: total - 4Gb)
>> > >
>> > >> I have two storage questions. How big can any one field in one doc be?
>> > >> Also, is there any limit on the size of a CouchDB? Meaning, if I have
>> > 1TB
>> > >> of free disk space, will the db grow to 1TB in size?
>> > >>
>> > >
>> >
>>
Re: Storage limitations?
Posted by Mark Hahn <ma...@reevuit.com>.
> sorting/displaying a comments thread with replies by appending the replies
within their respective original comment field.
Wouldn't it be more natural to use an array? And it would be more flexible
for possible future changes. It doesn't affect the size of the field but
there is no way a comment thread is going to get up to even a megabyte.
The entire bible fits in a megabyte.
On Sat, Nov 9, 2013 at 9:22 AM, Mark Deibert <ma...@gmail.com> wrote:
> Ok I was assuming the db could be really big. I'm storing images so it
> _will_ get pretty large.
>
> For the field size question: I had an idea to greatly simplify
> sorting/displaying a comments thread with replies by appending the replies
> within their respective original comment field. Since the order I want to
> display the comments is the natural chronological order, a simple append
> process seems to make sense. Haven't tried yet but seems logical.
>
>
> On Sat, Nov 9, 2013 at 11:32 AM, Robert Newson <rn...@apache.org> wrote:
>
> > while the default configuration has max_document_size as 4Gb, it would
> > be very unwise to make a document anywhere near that large. Database
> > size is limited only by disk capacity but note that you'll need some
> > free space for compacting the database.
> >
> > On 9 November 2013 15:28, first-leon@yandex.ru <fi...@yandex.ru>
> > wrote:
> > > Base size in no limit. But file system may be limit by file size.
> > > Doc size limit is 4Gb. (one or two or more field: total - 4Gb)
> > >
> > >> I have two storage questions. How big can any one field in one doc be?
> > >> Also, is there any limit on the size of a CouchDB? Meaning, if I have
> > 1TB
> > >> of free disk space, will the db grow to 1TB in size?
> > >>
> > >
> >
>
Re: Storage limitations?
Posted by Mark Deibert <ma...@gmail.com>.
Ok I was assuming the db could be really big. I'm storing images so it
_will_ get pretty large.
For the field size question: I had an idea to greatly simplify
sorting/displaying a comments thread with replies by appending the replies
within their respective original comment field. Since the order I want to
display the comments is the natural chronological order, a simple append
process seems to make sense. Haven't tried yet but seems logical.
On Sat, Nov 9, 2013 at 11:32 AM, Robert Newson <rn...@apache.org> wrote:
> while the default configuration has max_document_size as 4Gb, it would
> be very unwise to make a document anywhere near that large. Database
> size is limited only by disk capacity but note that you'll need some
> free space for compacting the database.
>
> On 9 November 2013 15:28, first-leon@yandex.ru <fi...@yandex.ru>
> wrote:
> > Base size in no limit. But file system may be limit by file size.
> > Doc size limit is 4Gb. (one or two or more field: total - 4Gb)
> >
> >> I have two storage questions. How big can any one field in one doc be?
> >> Also, is there any limit on the size of a CouchDB? Meaning, if I have
> 1TB
> >> of free disk space, will the db grow to 1TB in size?
> >>
> >
>
Re: Storage limitations?
Posted by Robert Newson <rn...@apache.org>.
while the default configuration has max_document_size as 4Gb, it would
be very unwise to make a document anywhere near that large. Database
size is limited only by disk capacity but note that you'll need some
free space for compacting the database.
On 9 November 2013 15:28, first-leon@yandex.ru <fi...@yandex.ru> wrote:
> Base size in no limit. But file system may be limit by file size.
> Doc size limit is 4Gb. (one or two or more field: total - 4Gb)
>
>> I have two storage questions. How big can any one field in one doc be?
>> Also, is there any limit on the size of a CouchDB? Meaning, if I have 1TB
>> of free disk space, will the db grow to 1TB in size?
>>
>
Re: Storage limitations?
Posted by "first-leon@yandex.ru" <fi...@yandex.ru>.
Base size in no limit. But file system may be limit by file size.
Doc size limit is 4Gb. (one or two or more field: total - 4Gb)
> I have two storage questions. How big can any one field in one doc be?
> Also, is there any limit on the size of a CouchDB? Meaning, if I have 1TB
> of free disk space, will the db grow to 1TB in size?
>