You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Chris Anderson <jc...@apache.org> on 2009/03/18 15:59:21 UTC
Re: Trying to see if CouchDB is right for our project...
On Wed, Mar 18, 2009 at 7:44 AM, Dean Landolt <de...@deanlandolt.com> wrote:
> On Wed, Mar 18, 2009 at 1:49 AM, Daniel Friesen <da...@telus.net>wrote:
>
>> Ok, I started a document here:
>> http://wiki.apache.org/couchdb/Forward_document_references
>>
>> There is a little more of the document to write. And I don't get how to
>> format the code so it isn't messed up.
>> I'll ping the dev list when I get back.
>
>
> Interesting ideas. One thing I'd add is that if this were implemented it
> would probably be best to make use of json referencing [1] which would allow
> you to make references by id or path.
>
> [1] http://www.json.com/2007/10/19/json-referencing-proposal-and-library/
>
I like the notion of applying this to views rather than documents, as
it is more general purpose. It's important that query techniques not
be tied to particular document schemas. Maybe the simplest way it
could work (start with simple first, then complete...) is to have the
view row (key, value) structure be:
(key, [array of keys to drill into])
And then query with an option for how far to drill. This version of it
does not have room for a payload (other than the docids) so maybe it's
worth forcing the array into a field on the value, so
(key, {"subkeys" : [array of keys to drill into], "foo" : "bar"})
Something like json-referencing could give us the ability to make the
value-schema flexible as well, but that's not core to the basic
question about recursive view traversal.
The implementation is only successful if the drill-down can happen in
nearly constant memory space (so any gatherings would have to be
accomplished in a reduce-like phase, which we can consider out of
scope for now...). Anyone want to hazard a guess as to the big-O
performance of a beast like this?
The basic memory bloat of buffering all the values can be avoided by
echoing them to the client and forgetting them. The only residual
bloat I see is the stack, which could become quite deep if depth is
unlimited. Another complication is circular references.
If the implementation is correct I can see this as a good plugin for
CouchDB. Once it's written all you need is a bunch of people to use it
and like it, and it could be a solid candidate for inclusion in core.
This thread dearly belongs on dev@ - cross posting this message, let's
try to keep subsequent replies on dev@.
Cheers,
Chris
--
Chris Anderson
http://jchris.mfdz.com
Re: Trying to see if CouchDB is right for our project...
Posted by Daniel Friesen <da...@telus.net>.
I did have another idea to propose, haven't got a name for it yet. I
just wanted to start with the simplest idea, this one required very
little modification to CouchDB and would work in a pretty efficient way
that in some small way could even be indexed for better performance
(theoretically if the jsonParse being used supported the second callback
function for mutating data, it might be possible to use that to index
_document tags on save at the same time as parsing the json)
Views are basically pieces of code for creating indexes so they only
operate on a single document of data. So I was thinking of a method of
querying for data server side and outputting it. _query if you will.
((pardon if I'm getting the general api path naming idea wrong))
/database/_query/queryname?param[key]=foo
/database/document/_query/queryname?param[key]=foo
db is the couch db, doc the doc if called in /database/document/...
fashion, and params is the param data send in the url.
/database/_query/jits?param[root]=id
/database/id/_query/jits?param[root]=id
query: function(db, doc, params) {
var root = doc || db.get(params[root]);
var hash = {};
var jitQue = [root];
var jit;
while(jit = jitQue.shift()) {
hash[jit] = db.get(jit);
var c = hash[jit].children;
for ( var i = 0, l = c.length; i<l; i++ ) {
if( hash[c[i]] ) continue;
jitQue.push(c[i]);
}
}
return hash;
}
The db would allow for any type of access, be it directly to a document,
or to a view. The ideal purpose would be only used when you have
document ids you already know about (or view keys) and want to grab the
data.
Another possible name might be _aggregate.
~Daniel Friesen (Dantman, Nadir-Seen-Fire)
Chris Anderson wrote:
> On Wed, Mar 18, 2009 at 7:44 AM, Dean Landolt <de...@deanlandolt.com> wrote:
>
>> On Wed, Mar 18, 2009 at 1:49 AM, Daniel Friesen <da...@telus.net>wrote:
>>
>>
>>> Ok, I started a document here:
>>> http://wiki.apache.org/couchdb/Forward_document_references
>>>
>>> There is a little more of the document to write. And I don't get how to
>>> format the code so it isn't messed up.
>>> I'll ping the dev list when I get back.
>>>
>> Interesting ideas. One thing I'd add is that if this were implemented it
>> would probably be best to make use of json referencing [1] which would allow
>> you to make references by id or path.
>>
>> [1] http://www.json.com/2007/10/19/json-referencing-proposal-and-library/
>>
>>
>
> I like the notion of applying this to views rather than documents, as
> it is more general purpose. It's important that query techniques not
> be tied to particular document schemas. Maybe the simplest way it
> could work (start with simple first, then complete...) is to have the
> view row (key, value) structure be:
>
> (key, [array of keys to drill into])
>
> And then query with an option for how far to drill. This version of it
> does not have room for a payload (other than the docids) so maybe it's
> worth forcing the array into a field on the value, so
>
> (key, {"subkeys" : [array of keys to drill into], "foo" : "bar"})
>
> Something like json-referencing could give us the ability to make the
> value-schema flexible as well, but that's not core to the basic
> question about recursive view traversal.
>
> The implementation is only successful if the drill-down can happen in
> nearly constant memory space (so any gatherings would have to be
> accomplished in a reduce-like phase, which we can consider out of
> scope for now...). Anyone want to hazard a guess as to the big-O
> performance of a beast like this?
>
> The basic memory bloat of buffering all the values can be avoided by
> echoing them to the client and forgetting them. The only residual
> bloat I see is the stack, which could become quite deep if depth is
> unlimited. Another complication is circular references.
>
> If the implementation is correct I can see this as a good plugin for
> CouchDB. Once it's written all you need is a bunch of people to use it
> and like it, and it could be a solid candidate for inclusion in core.
>
> This thread dearly belongs on dev@ - cross posting this message, let's
> try to keep subsequent replies on dev@.
>
> Cheers,
> Chris
>
>
>
Re: Trying to see if CouchDB is right for our project...
Posted by Daniel Friesen <li...@danielfriesen.name>.
((Resending this to the dev list since I've been having issues with my
old email attached to gmane))
I did have another idea to propose, haven't got a name for it yet. I
just wanted to start with the simplest idea, this one required very
little modification to CouchDB and would work in a pretty efficient way
that in some small way could even be indexed for better performance
(theoretically if the jsonParse being used supported the second callback
function for mutating data, it might be possible to use that to index
_document tags on save at the same time as parsing the json)
Views are basically pieces of code for creating indexes so they only
operate on a single document of data. So I was thinking of a method of
querying for data server side and outputting it. _query if you will.
((pardon if I'm getting the general api path naming idea wrong))
/database/_query/queryname?param[key]=foo
/database/document/_query/queryname?param[key]=foo
db is the couch db, doc the doc if called in /database/document/...
fashion, and params is the param data send in the url.
/database/_query/jits?param[root]=id
/database/id/_query/jits?param[root]=id
query: function(db, doc, params) {
var root = doc || db.get(params[root]);
var hash = {};
var jitQue = [root];
var jit;
while(jit = jitQue.shift()) {
hash[jit] = db.get(jit);
var c = hash[jit].children;
for ( var i = 0, l = c.length; i<l; i++ ) {
if( hash[c[i]] ) continue;
jitQue.push(c[i]);
}
}
return hash;
}
The db would allow for any type of access, be it directly to a document,
or to a view. The ideal purpose would be only used when you have
document ids you already know about (or view keys) and want to grab the
data.
Another possible name might be _aggregate.
~Daniel Friesen (Dantman, Nadir-Seen-Fire)
Chris Anderson wrote:
> On Wed, Mar 18, 2009 at 7:44 AM, Dean Landolt <de...@deanlandolt.com> wrote:
>
>> On Wed, Mar 18, 2009 at 1:49 AM, Daniel Friesen <da...@telus.net>wrote:
>>
>>
>>> Ok, I started a document here:
>>> http://wiki.apache.org/couchdb/Forward_document_references
>>>
>>> There is a little more of the document to write. And I don't get how to
>>> format the code so it isn't messed up.
>>> I'll ping the dev list when I get back.
>>>
>> Interesting ideas. One thing I'd add is that if this were implemented it
>> would probably be best to make use of json referencing [1] which would allow
>> you to make references by id or path.
>>
>> [1] http://www.json.com/2007/10/19/json-referencing-proposal-and-library/
>>
>>
>
> I like the notion of applying this to views rather than documents, as
> it is more general purpose. It's important that query techniques not
> be tied to particular document schemas. Maybe the simplest way it
> could work (start with simple first, then complete...) is to have the
> view row (key, value) structure be:
>
> (key, [array of keys to drill into])
>
> And then query with an option for how far to drill. This version of it
> does not have room for a payload (other than the docids) so maybe it's
> worth forcing the array into a field on the value, so
>
> (key, {"subkeys" : [array of keys to drill into], "foo" : "bar"})
>
> Something like json-referencing could give us the ability to make the
> value-schema flexible as well, but that's not core to the basic
> question about recursive view traversal.
>
> The implementation is only successful if the drill-down can happen in
> nearly constant memory space (so any gatherings would have to be
> accomplished in a reduce-like phase, which we can consider out of
> scope for now...). Anyone want to hazard a guess as to the big-O
> performance of a beast like this?
>
> The basic memory bloat of buffering all the values can be avoided by
> echoing them to the client and forgetting them. The only residual
> bloat I see is the stack, which could become quite deep if depth is
> unlimited. Another complication is circular references.
>
> If the implementation is correct I can see this as a good plugin for
> CouchDB. Once it's written all you need is a bunch of people to use it
> and like it, and it could be a solid candidate for inclusion in core.
>
> This thread dearly belongs on dev@ - cross posting this message, let's
> try to keep subsequent replies on dev@.
>
> Cheers,
> Chris
>
>
>