Some CouchDB internals questions?

[Zdravko Gligic <zg...@gmail.com>]

Folks,

I created a sample proof of concept app and it created more questions
than it answered with regard to CouchDB and resource usage.  From the
best that I can tell, some 1,500 docs which are about 500 characters
in size (in terms of readable text) to maybe 1k at most are being
bloated to an average of 10k per doc - with just 2 'plain' views and a
3rd view that re-emits the doc as well.

So, is there any literature around that explains at least a bit some
of the CouchDB internals, in terms of at least disk usage.  How much
might each byte end up taking to store (with whatever built-in
overhead) in a doc and what about views and indexing? Are there any
even very generalized formulas like multiply all bytes in index by 2
and add 10 bytes for each filed/attribute in index and then add 1k if
you are building indices on a cloudy day ;?)

Thanks for your help,
Zdravko

Re: Some CouchDB internals questions?

[Dominic Tarr <do...@gmail.com>]

do you know about the include_doc option on view queries?

if you emit(key, doc) in the view, then the whole doc will be saved in the
view cache,
but if you query a view with the include_doc=true then the view results will
include the whole doc.
this is fetched from the db and not stored in the view.

this will slow reads, but use less space.
which option is best depends on your application.

it's mentioned here:

http://wiki.apache.org/couchdb/HTTP_view_API

On Thu, Mar 17, 2011 at 5:53 AM, Zdravko Gligic <zg...@gmail.com> wrote:

> WOW !
>
> So, how long might it take for this not only to become part of CouchDB
> core but then also to get implemented by all of the ohter CouchDB
> dialects such as CouchBase and BigCouch ,etc ?
>
> And as dumb as it might sound ;) why was this not done (: the right
> way :) from the very beginning ;?)
>
> On Wed, Mar 16, 2011 at 10:02 AM, Filipe David Manana
> <fd...@apache.org> wrote:
> > Zdravko,
> >
> > Yesterday a performance related ticket was created:
> >
> > https://issues.apache.org/jira/browse/COUCHDB-1092
> >
> > Apart from the performance improvements, it also reduces very
> > significantly the database sizes (from 2 times less to about 10 times
> > less). So you might be interested to follow/read.
> >
> > On Tue, Mar 15, 2011 at 7:32 PM, Paul Davis <pa...@gmail.com>
> wrote:
> >> On Tue, Mar 15, 2011 at 2:53 PM, Zdravko Gligic <zg...@gmail.com>
> wrote:
> >>>> Have you compacted your db and views?
> >>>
> >>> Yes
> >>>
> >>>> There's unfortunately no direct way to calculate a upper threshold, it
> >>>> really depends on your method for inserting as well as how often you
> >>>> compact.
> >>>
> >>> Once both (docs and view) are compacted, is the resulting size at all
> >>> dependent on how the docs and/or views were created in the first place
> >>> (one at a time or in bulk or whatever) ?
> >>>
> >>
> >> I think to get the absolute minimum post-compaction size you need to
> >> compact twice. I haven't done lots of extensive testing on this, but
> >> last I recall the basic logic was the first time can end up writing
> >> docs in a somewhat randomish ordering depending on how they were
> >> inserted.
> >>
> >>>> This is due to the tail append storage which will orphan data
> >>>> in the file as it writes new records to the various internal data
> >>>> structures.
> >>>
> >>> My 1,500 docs are taking up almost 15 meg (roughly 1/4-1k docs with 2
> >>> views + 1 view with doc re-emit) and I believe were around 50meg
> >>> before compactions.
> >>>
> >>
> >> More importantly, what was the datasize post-compaction though? If
> >> your main db is 15Meg, and you have a view that re-emits the doc, I'd
> >> expect you to have a total size of at least 30Meg. Depending on what
> >> you're emitting in the other two views getting closer to that 50 isn't
> >> hugely out of the question.
> >>
> >
> >
> >
> > --
> > Filipe David Manana,
> > fdmanana@gmail.com, fdmanana@apache.org
> >
> > "Reasonable men adapt themselves to the world.
> >  Unreasonable men adapt the world to themselves.
> >  That's why all progress depends on unreasonable men."
> >
>

Re: Some CouchDB internals questions?

[Robert Newson <ro...@gmail.com>]

I'd like to see more discussion on this patch before it goes in, I
think we're not quite at consensus yet. Given the scope of the change,
I think we need that.

I'd also agree with Filipe that this is "get it right *then* make it
fast" in action, though davisp raises concerns on the idea in general.

I read most of the patch and liked much of it (the bulk is just
calling the new fun instead of the old, which is unobjectionable), but
I'd like to hear Paul's concerns again and see if we can address them
satisfactorily.

B.

On 17 March 2011 08:56, Filipe David Manana <fd...@apache.org> wrote:
> On Wed, Mar 16, 2011 at 4:53 PM, Zdravko Gligic <zg...@gmail.com> wrote:
>> WOW !
>>
>> So, how long might it take for this not only to become part of CouchDB
>> core but then also to get implemented by all of the ohter CouchDB
>> dialects such as CouchBase and BigCouch ,etc ?
>
> Hopefully shouldn't take long to land into Apache CouchDB's trunk, as
> so far none of the existing features/components are affected with a
> performance drop, and neither it changes any API. The change itself is
> relatively simple as well.
>
> For Bigcouch, I can't tell, you should ask Cloudant people (Adam, Robert, etc).
>
>>
>> And as dumb as it might sound ;) why was this not done (: the right
>> way :) from the very beginning ;?)
>
> Have no idea. I've only been in the community for little more than 1 year.
> I would assume that initially developers were more worried about
> correctness and elegant APIs, which is perfectly reasonable and sane -
> performance should come after.
>
> regards
>
>>
>> On Wed, Mar 16, 2011 at 10:02 AM, Filipe David Manana
>> <fd...@apache.org> wrote:
>>> Zdravko,
>>>
>>> Yesterday a performance related ticket was created:
>>>
>>> https://issues.apache.org/jira/browse/COUCHDB-1092
>>>
>>> Apart from the performance improvements, it also reduces very
>>> significantly the database sizes (from 2 times less to about 10 times
>>> less). So you might be interested to follow/read.
>>>
>>> On Tue, Mar 15, 2011 at 7:32 PM, Paul Davis <pa...@gmail.com> wrote:
>>>> On Tue, Mar 15, 2011 at 2:53 PM, Zdravko Gligic <zg...@gmail.com> wrote:
>>>>>> Have you compacted your db and views?
>>>>>
>>>>> Yes
>>>>>
>>>>>> There's unfortunately no direct way to calculate a upper threshold, it
>>>>>> really depends on your method for inserting as well as how often you
>>>>>> compact.
>>>>>
>>>>> Once both (docs and view) are compacted, is the resulting size at all
>>>>> dependent on how the docs and/or views were created in the first place
>>>>> (one at a time or in bulk or whatever) ?
>>>>>
>>>>
>>>> I think to get the absolute minimum post-compaction size you need to
>>>> compact twice. I haven't done lots of extensive testing on this, but
>>>> last I recall the basic logic was the first time can end up writing
>>>> docs in a somewhat randomish ordering depending on how they were
>>>> inserted.
>>>>
>>>>>> This is due to the tail append storage which will orphan data
>>>>>> in the file as it writes new records to the various internal data
>>>>>> structures.
>>>>>
>>>>> My 1,500 docs are taking up almost 15 meg (roughly 1/4-1k docs with 2
>>>>> views + 1 view with doc re-emit) and I believe were around 50meg
>>>>> before compactions.
>>>>>
>>>>
>>>> More importantly, what was the datasize post-compaction though? If
>>>> your main db is 15Meg, and you have a view that re-emits the doc, I'd
>>>> expect you to have a total size of at least 30Meg. Depending on what
>>>> you're emitting in the other two views getting closer to that 50 isn't
>>>> hugely out of the question.
>>>>
>>>
>>>
>>>
>>> --
>>> Filipe David Manana,
>>> fdmanana@gmail.com, fdmanana@apache.org
>>>
>>> "Reasonable men adapt themselves to the world.
>>>  Unreasonable men adapt the world to themselves.
>>>  That's why all progress depends on unreasonable men."
>>>
>>
>
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com, fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
>  Unreasonable men adapt the world to themselves.
>  That's why all progress depends on unreasonable men."
>

Re: Some CouchDB internals questions?

[Filipe David Manana <fd...@apache.org>]

On Wed, Mar 16, 2011 at 4:53 PM, Zdravko Gligic <zg...@gmail.com> wrote:
> WOW !
>
> So, how long might it take for this not only to become part of CouchDB
> core but then also to get implemented by all of the ohter CouchDB
> dialects such as CouchBase and BigCouch ,etc ?

Hopefully shouldn't take long to land into Apache CouchDB's trunk, as
so far none of the existing features/components are affected with a
performance drop, and neither it changes any API. The change itself is
relatively simple as well.

For Bigcouch, I can't tell, you should ask Cloudant people (Adam, Robert, etc).

>
> And as dumb as it might sound ;) why was this not done (: the right
> way :) from the very beginning ;?)

Have no idea. I've only been in the community for little more than 1 year.
I would assume that initially developers were more worried about
correctness and elegant APIs, which is perfectly reasonable and sane -
performance should come after.

regards

>
> On Wed, Mar 16, 2011 at 10:02 AM, Filipe David Manana
> <fd...@apache.org> wrote:
>> Zdravko,
>>
>> Yesterday a performance related ticket was created:
>>
>> https://issues.apache.org/jira/browse/COUCHDB-1092
>>
>> Apart from the performance improvements, it also reduces very
>> significantly the database sizes (from 2 times less to about 10 times
>> less). So you might be interested to follow/read.
>>
>> On Tue, Mar 15, 2011 at 7:32 PM, Paul Davis <pa...@gmail.com> wrote:
>>> On Tue, Mar 15, 2011 at 2:53 PM, Zdravko Gligic <zg...@gmail.com> wrote:
>>>>> Have you compacted your db and views?
>>>>
>>>> Yes
>>>>
>>>>> There's unfortunately no direct way to calculate a upper threshold, it
>>>>> really depends on your method for inserting as well as how often you
>>>>> compact.
>>>>
>>>> Once both (docs and view) are compacted, is the resulting size at all
>>>> dependent on how the docs and/or views were created in the first place
>>>> (one at a time or in bulk or whatever) ?
>>>>
>>>
>>> I think to get the absolute minimum post-compaction size you need to
>>> compact twice. I haven't done lots of extensive testing on this, but
>>> last I recall the basic logic was the first time can end up writing
>>> docs in a somewhat randomish ordering depending on how they were
>>> inserted.
>>>
>>>>> This is due to the tail append storage which will orphan data
>>>>> in the file as it writes new records to the various internal data
>>>>> structures.
>>>>
>>>> My 1,500 docs are taking up almost 15 meg (roughly 1/4-1k docs with 2
>>>> views + 1 view with doc re-emit) and I believe were around 50meg
>>>> before compactions.
>>>>
>>>
>>> More importantly, what was the datasize post-compaction though? If
>>> your main db is 15Meg, and you have a view that re-emits the doc, I'd
>>> expect you to have a total size of at least 30Meg. Depending on what
>>> you're emitting in the other two views getting closer to that 50 isn't
>>> hugely out of the question.
>>>
>>
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana@gmail.com, fdmanana@apache.org
>>
>> "Reasonable men adapt themselves to the world.
>>  Unreasonable men adapt the world to themselves.
>>  That's why all progress depends on unreasonable men."
>>
>



-- 
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: Some CouchDB internals questions?

[Zdravko Gligic <zg...@gmail.com>]

WOW !

So, how long might it take for this not only to become part of CouchDB
core but then also to get implemented by all of the ohter CouchDB
dialects such as CouchBase and BigCouch ,etc ?

And as dumb as it might sound ;) why was this not done (: the right
way :) from the very beginning ;?)

On Wed, Mar 16, 2011 at 10:02 AM, Filipe David Manana
<fd...@apache.org> wrote:
> Zdravko,
>
> Yesterday a performance related ticket was created:
>
> https://issues.apache.org/jira/browse/COUCHDB-1092
>
> Apart from the performance improvements, it also reduces very
> significantly the database sizes (from 2 times less to about 10 times
> less). So you might be interested to follow/read.
>
> On Tue, Mar 15, 2011 at 7:32 PM, Paul Davis <pa...@gmail.com> wrote:
>> On Tue, Mar 15, 2011 at 2:53 PM, Zdravko Gligic <zg...@gmail.com> wrote:
>>>> Have you compacted your db and views?
>>>
>>> Yes
>>>
>>>> There's unfortunately no direct way to calculate a upper threshold, it
>>>> really depends on your method for inserting as well as how often you
>>>> compact.
>>>
>>> Once both (docs and view) are compacted, is the resulting size at all
>>> dependent on how the docs and/or views were created in the first place
>>> (one at a time or in bulk or whatever) ?
>>>
>>
>> I think to get the absolute minimum post-compaction size you need to
>> compact twice. I haven't done lots of extensive testing on this, but
>> last I recall the basic logic was the first time can end up writing
>> docs in a somewhat randomish ordering depending on how they were
>> inserted.
>>
>>>> This is due to the tail append storage which will orphan data
>>>> in the file as it writes new records to the various internal data
>>>> structures.
>>>
>>> My 1,500 docs are taking up almost 15 meg (roughly 1/4-1k docs with 2
>>> views + 1 view with doc re-emit) and I believe were around 50meg
>>> before compactions.
>>>
>>
>> More importantly, what was the datasize post-compaction though? If
>> your main db is 15Meg, and you have a view that re-emits the doc, I'd
>> expect you to have a total size of at least 30Meg. Depending on what
>> you're emitting in the other two views getting closer to that 50 isn't
>> hugely out of the question.
>>
>
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com, fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
>  Unreasonable men adapt the world to themselves.
>  That's why all progress depends on unreasonable men."
>

Re: Some CouchDB internals questions?

[Filipe David Manana <fd...@apache.org>]

Zdravko,

Yesterday a performance related ticket was created:

https://issues.apache.org/jira/browse/COUCHDB-1092

Apart from the performance improvements, it also reduces very
significantly the database sizes (from 2 times less to about 10 times
less). So you might be interested to follow/read.

On Tue, Mar 15, 2011 at 7:32 PM, Paul Davis <pa...@gmail.com> wrote:
> On Tue, Mar 15, 2011 at 2:53 PM, Zdravko Gligic <zg...@gmail.com> wrote:
>>> Have you compacted your db and views?
>>
>> Yes
>>
>>> There's unfortunately no direct way to calculate a upper threshold, it
>>> really depends on your method for inserting as well as how often you
>>> compact.
>>
>> Once both (docs and view) are compacted, is the resulting size at all
>> dependent on how the docs and/or views were created in the first place
>> (one at a time or in bulk or whatever) ?
>>
>
> I think to get the absolute minimum post-compaction size you need to
> compact twice. I haven't done lots of extensive testing on this, but
> last I recall the basic logic was the first time can end up writing
> docs in a somewhat randomish ordering depending on how they were
> inserted.
>
>>> This is due to the tail append storage which will orphan data
>>> in the file as it writes new records to the various internal data
>>> structures.
>>
>> My 1,500 docs are taking up almost 15 meg (roughly 1/4-1k docs with 2
>> views + 1 view with doc re-emit) and I believe were around 50meg
>> before compactions.
>>
>
> More importantly, what was the datasize post-compaction though? If
> your main db is 15Meg, and you have a view that re-emits the doc, I'd
> expect you to have a total size of at least 30Meg. Depending on what
> you're emitting in the other two views getting closer to that 50 isn't
> hugely out of the question.
>



-- 
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: Some CouchDB internals questions?

[Paul Davis <pa...@gmail.com>]

On Tue, Mar 15, 2011 at 2:53 PM, Zdravko Gligic <zg...@gmail.com> wrote:
>> Have you compacted your db and views?
>
> Yes
>
>> There's unfortunately no direct way to calculate a upper threshold, it
>> really depends on your method for inserting as well as how often you
>> compact.
>
> Once both (docs and view) are compacted, is the resulting size at all
> dependent on how the docs and/or views were created in the first place
> (one at a time or in bulk or whatever) ?
>

I think to get the absolute minimum post-compaction size you need to
compact twice. I haven't done lots of extensive testing on this, but
last I recall the basic logic was the first time can end up writing
docs in a somewhat randomish ordering depending on how they were
inserted.

>> This is due to the tail append storage which will orphan data
>> in the file as it writes new records to the various internal data
>> structures.
>
> My 1,500 docs are taking up almost 15 meg (roughly 1/4-1k docs with 2
> views + 1 view with doc re-emit) and I believe were around 50meg
> before compactions.
>

More importantly, what was the datasize post-compaction though? If
your main db is 15Meg, and you have a view that re-emits the doc, I'd
expect you to have a total size of at least 30Meg. Depending on what
you're emitting in the other two views getting closer to that 50 isn't
hugely out of the question.

Re: Some CouchDB internals questions?

[Zdravko Gligic <zg...@gmail.com>]

> Have you compacted your db and views?

Yes

> There's unfortunately no direct way to calculate a upper threshold, it
> really depends on your method for inserting as well as how often you
> compact.

Once both (docs and view) are compacted, is the resulting size at all
dependent on how the docs and/or views were created in the first place
(one at a time or in bulk or whatever) ?

> This is due to the tail append storage which will orphan data
> in the file as it writes new records to the various internal data
> structures.

My 1,500 docs are taking up almost 15 meg (roughly 1/4-1k docs with 2
views + 1 view with doc re-emit) and I believe were around 50meg
before compactions.

Thanks Paul.

Re: Some CouchDB internals questions?

[Paul Davis <pa...@gmail.com>]

On Tue, Mar 15, 2011 at 2:37 PM, Zdravko Gligic <zg...@gmail.com> wrote:
> Folks,
>
> I created a sample proof of concept app and it created more questions
> than it answered with regard to CouchDB and resource usage.  From the
> best that I can tell, some 1,500 docs which are about 500 characters
> in size (in terms of readable text) to maybe 1k at most are being
> bloated to an average of 10k per doc - with just 2 'plain' views and a
> 3rd view that re-emits the doc as well.
>
> So, is there any literature around that explains at least a bit some
> of the CouchDB internals, in terms of at least disk usage.  How much
> might each byte end up taking to store (with whatever built-in
> overhead) in a doc and what about views and indexing? Are there any
> even very generalized formulas like multiply all bytes in index by 2
> and add 10 bytes for each filed/attribute in index and then add 1k if
> you are building indices on a cloudy day ;?)
>
> Thanks for your help,
> Zdravko
>

Have you compacted your db and views?

There's unfortunately no direct way to calculate a upper threshold, it
really depends on your method for inserting as well as how often you
compact. This is due to the tail append storage which will orphan data
in the file as it writes new records to the various internal data
structures.