Newbie: Locality of Reference in Couch

[Robert Buck <bu...@gmail.com>]

Hi Folks,

Thank you for kindly answering my last round of questions. Here is
another question related to Couch:

What sort of locality of reference exists in Couch with respect to
retrieval of state ? Is locality of reference solely at the document
level, or is locality of reference also exhibited elsewhere that
developers can take advantage of ?

Thanks in advance,

Bob

Re: Newbie: Locality of Reference in Couch

[Robert Buck <bu...@gmail.com>]

On Fri, May 28, 2010 at 6:10 PM, J Chris Anderson <jc...@gmail.com> wrote:
>
> On May 27, 2010, at 3:16 AM, Robert Buck wrote:
>
>> On Wed, May 26, 2010 at 9:01 PM, Randall Leeds <ra...@gmail.com> wrote:
>>> On Wed, May 26, 2010 at 15:39, J Chris Anderson <jc...@gmail.com> wrote:
>>>>
>>>> On May 26, 2010, at 1:53 PM, Robert Buck wrote:
>>>>
>>>>> Hi Folks,
>>>>>
>>>>> Thank you for kindly answering my last round of questions. Here is
>>>>> another question related to Couch:
>>>>>
>>>>> What sort of locality of reference exists in Couch with respect to
>>>>> retrieval of state ? Is locality of reference solely at the document
>>>>> level, or is locality of reference also exhibited elsewhere that
>>>>> developers can take advantage of ?
>>>>>
>>>>
>>>> Hmm, I'm not sure what you mean by locality of reference (I know it in the context of performance optimizations)
>>>
>>> Same.
>>>
>>> If you're using it this way, I guess the best answer is that documents
>>> might be very sparsely spread out across the disk in a large database
>>> that has not been compacted for a long time. Documents updated around
>>> the same time might be closer to the tail of the file. We really rely
>>> on the filesystem cache to make this something we can forget about.
>>>
>>> Does that answer anything at all?
>>
>> That's good, that confirms what I have read, just wanted to verify.
>> Some database technologies allow you to "group" data more closely
>> together, some call this a container, others call it a segment. From
>> what I read Couch apparently has no such feature.
>>
>> Thanks so much for your help.
>
> Another performance thing you can do it, is use document ids that are near each other in the keyspace. This will give better cache-locality in the filesystem cache. Luckily, CouchDB does a good job of this by default, if you use the built-in uuid generator.
>
> There were some analysis / benchmarks on this list a few months ago about the difference between sorted, sorry I don't have the links handy. Upshot, small, sequential-ish ids are the fastest.
>
> Chris

Thanks

Re: Newbie: Locality of Reference in Couch

[J Chris Anderson <jc...@gmail.com>]

On May 27, 2010, at 3:16 AM, Robert Buck wrote:

> On Wed, May 26, 2010 at 9:01 PM, Randall Leeds <ra...@gmail.com> wrote:
>> On Wed, May 26, 2010 at 15:39, J Chris Anderson <jc...@gmail.com> wrote:
>>> 
>>> On May 26, 2010, at 1:53 PM, Robert Buck wrote:
>>> 
>>>> Hi Folks,
>>>> 
>>>> Thank you for kindly answering my last round of questions. Here is
>>>> another question related to Couch:
>>>> 
>>>> What sort of locality of reference exists in Couch with respect to
>>>> retrieval of state ? Is locality of reference solely at the document
>>>> level, or is locality of reference also exhibited elsewhere that
>>>> developers can take advantage of ?
>>>> 
>>> 
>>> Hmm, I'm not sure what you mean by locality of reference (I know it in the context of performance optimizations)
>> 
>> Same.
>> 
>> If you're using it this way, I guess the best answer is that documents
>> might be very sparsely spread out across the disk in a large database
>> that has not been compacted for a long time. Documents updated around
>> the same time might be closer to the tail of the file. We really rely
>> on the filesystem cache to make this something we can forget about.
>> 
>> Does that answer anything at all?
> 
> That's good, that confirms what I have read, just wanted to verify.
> Some database technologies allow you to "group" data more closely
> together, some call this a container, others call it a segment. From
> what I read Couch apparently has no such feature.
> 
> Thanks so much for your help.

Another performance thing you can do it, is use document ids that are near each other in the keyspace. This will give better cache-locality in the filesystem cache. Luckily, CouchDB does a good job of this by default, if you use the built-in uuid generator.

There were some analysis / benchmarks on this list a few months ago about the difference between sorted, sorry I don't have the links handy. Upshot, small, sequential-ish ids are the fastest.

Chris

Re: Newbie: Locality of Reference in Couch

[Robert Buck <bu...@gmail.com>]

On Wed, May 26, 2010 at 9:01 PM, Randall Leeds <ra...@gmail.com> wrote:
> On Wed, May 26, 2010 at 15:39, J Chris Anderson <jc...@gmail.com> wrote:
>>
>> On May 26, 2010, at 1:53 PM, Robert Buck wrote:
>>
>>> Hi Folks,
>>>
>>> Thank you for kindly answering my last round of questions. Here is
>>> another question related to Couch:
>>>
>>> What sort of locality of reference exists in Couch with respect to
>>> retrieval of state ? Is locality of reference solely at the document
>>> level, or is locality of reference also exhibited elsewhere that
>>> developers can take advantage of ?
>>>
>>
>> Hmm, I'm not sure what you mean by locality of reference (I know it in the context of performance optimizations)
>
> Same.
>
> If you're using it this way, I guess the best answer is that documents
> might be very sparsely spread out across the disk in a large database
> that has not been compacted for a long time. Documents updated around
> the same time might be closer to the tail of the file. We really rely
> on the filesystem cache to make this something we can forget about.
>
> Does that answer anything at all?

That's good, that confirms what I have read, just wanted to verify.
Some database technologies allow you to "group" data more closely
together, some call this a container, others call it a segment. From
what I read Couch apparently has no such feature.

Thanks so much for your help.

Re: Newbie: Locality of Reference in Couch

[Randall Leeds <ra...@gmail.com>]

On Wed, May 26, 2010 at 15:39, J Chris Anderson <jc...@gmail.com> wrote:
>
> On May 26, 2010, at 1:53 PM, Robert Buck wrote:
>
>> Hi Folks,
>>
>> Thank you for kindly answering my last round of questions. Here is
>> another question related to Couch:
>>
>> What sort of locality of reference exists in Couch with respect to
>> retrieval of state ? Is locality of reference solely at the document
>> level, or is locality of reference also exhibited elsewhere that
>> developers can take advantage of ?
>>
>
> Hmm, I'm not sure what you mean by locality of reference (I know it in the context of performance optimizations)

Same.

If you're using it this way, I guess the best answer is that documents
might be very sparsely spread out across the disk in a large database
that has not been compacted for a long time. Documents updated around
the same time might be closer to the tail of the file. We really rely
on the filesystem cache to make this something we can forget about.

Does that answer anything at all?

Re: Newbie: Locality of Reference in Couch

[J Chris Anderson <jc...@gmail.com>]

On May 26, 2010, at 1:53 PM, Robert Buck wrote:

> Hi Folks,
> 
> Thank you for kindly answering my last round of questions. Here is
> another question related to Couch:
> 
> What sort of locality of reference exists in Couch with respect to
> retrieval of state ? Is locality of reference solely at the document
> level, or is locality of reference also exhibited elsewhere that
> developers can take advantage of ?
> 

Hmm, I'm not sure what you mean by locality of reference (I know it in the context of performance optimizations)

But I have a feeling I know what you mean about state and documents.

Yes, there is another endpoint you can use, that gives you a comprehensive view of the state at a database level.

The _changes feed is an http comet feed of updates as they occur on the database.

http://books.couchdb.org/relax/reference/change-notifications

Also, note that the map reduce view indexes are built from MVCC snapshots of the database, so they are pure functional transforms of the database state as well.

Also note that the _show and _list functions are functional transforms from documents and views, so they are also based strictly on the database state.

Hope that helps.

Chris

> Thanks in advance,
> 
> Bob