The 1.0 Thread

[Damien Katz <da...@apache.org>]

Okay, time to ask the question, what features do we need to get to 1.0?

I'm going to list my must haves, and my nice to haves.

Must have:
- Document integrity checking: Using some sort of hashing scheme for  
end to end integrity checking of documents and attachments. Reusing  
the revision ID as the hash of the document might work, and has the  
benefit of allowing writing the same changes to 2 different servers  
and not causing a conflict. Also multiple clients can write the same  
change to a document and not get unnecessary conflicts.
- Reader/Writer access databases and servers: Allow/disallow  
anonymous, users, groups.
- Continuous replication: Keeping a constant connection and being able  
to replicate changes as soon as they happen.
- Better testing: We need really some performance and stress testing  
as part of the source. And we need much better code coverage in  
general with the testing.


Nice to have:
- Hashing/CRC everything written to disk, data, metadata, index  
structures, etc. But optional, since many filesystems actively  
integrity-check disk data.
- Better full text integration: Out of the box integration and the  
ability intersect results with views, for easier result formatting.  
Lucene would be the primary FT engine, but we make it pluggable, much  
like the view engines are.
- Attachment level replication: By tracking the revision when an  
attachment was modified, the replicator can avoid copying unchanged  
attachments to the target. The same can apply to json fields, but it's  
much less of a win there.
- Partitioning/sharding support: Ideally would be nice to have  
something that "just works" without a lot of setup.
- Built-in authentication: A plug-in that authenticates HTTP users and  
assign them roles. It would use a couch database as a directory that  
contains users documents, etc.
- Selective replication: The ability to replicate a subset of  
documents, using a javascript function as a selector.
- Server side doc processing: The ability to POST data and have  
arbitrary server-side processing. The simplest case is posting a  
document to a Js handler that can do some data cleanup and add default  
values the document before saving it. But ideally would be able to  
interact with the full database
- Scheduled replication: The ability to schedule replication every so  
often, like a cron job. But this can be done with an actual cron job  
and CURL, so it's not critical to have it built-in.

There are probably a bunch of things I forgot about.

Respond to this with your must haves and nice to haves. No promises  
you'll get your way (no guarantee for me for that matter), but lets  
start talking about it.

And anyone who wants to take on any of these issues: mine, yours or  
anyone else's, just do it. Read code, mail dev@ with questions and  
advice, write some code, repeat.

-Damien

Re: The 1.0 Thread

[Nathan Stott <nr...@gmail.com>]

Must have:eaccess issue fixed for Windows users

Nice to have:
Windows installer, Mark Hammond and I are working hard on this one :)

On Thu, Jun 18, 2009 at 4:34 PM, Damien Katz <da...@apache.org> wrote:

> Okay, time to ask the question, what features do we need to get to 1.0?
>
> I'm going to list my must haves, and my nice to haves.
>
> Must have:
> - Document integrity checking: Using some sort of hashing scheme for end to
> end integrity checking of documents and attachments. Reusing the revision ID
> as the hash of the document might work, and has the benefit of allowing
> writing the same changes to 2 different servers and not causing a conflict.
> Also multiple clients can write the same change to a document and not get
> unnecessary conflicts.
> - Reader/Writer access databases and servers: Allow/disallow anonymous,
> users, groups.
> - Continuous replication: Keeping a constant connection and being able to
> replicate changes as soon as they happen.
> - Better testing: We need really some performance and stress testing as
> part of the source. And we need much better code coverage in general with
> the testing.
>
>
> Nice to have:
> - Hashing/CRC everything written to disk, data, metadata, index structures,
> etc. But optional, since many filesystems actively integrity-check disk
> data.
> - Better full text integration: Out of the box integration and the ability
> intersect results with views, for easier result formatting. Lucene would be
> the primary FT engine, but we make it pluggable, much like the view engines
> are.
> - Attachment level replication: By tracking the revision when an attachment
> was modified, the replicator can avoid copying unchanged attachments to the
> target. The same can apply to json fields, but it's much less of a win
> there.
> - Partitioning/sharding support: Ideally would be nice to have something
> that "just works" without a lot of setup.
> - Built-in authentication: A plug-in that authenticates HTTP users and
> assign them roles. It would use a couch database as a directory that
> contains users documents, etc.
> - Selective replication: The ability to replicate a subset of documents,
> using a javascript function as a selector.
> - Server side doc processing: The ability to POST data and have arbitrary
> server-side processing. The simplest case is posting a document to a Js
> handler that can do some data cleanup and add default values the document
> before saving it. But ideally would be able to interact with the full
> database
> - Scheduled replication: The ability to schedule replication every so
> often, like a cron job. But this can be done with an actual cron job and
> CURL, so it's not critical to have it built-in.
>
> There are probably a bunch of things I forgot about.
>
> Respond to this with your must haves and nice to haves. No promises you'll
> get your way (no guarantee for me for that matter), but lets start talking
> about it.
>
> And anyone who wants to take on any of these issues: mine, yours or anyone
> else's, just do it. Read code, mail dev@ with questions and advice, write
> some code, repeat.
>
> -Damien
>

Re: The 1.0 Thread

[Paul Davis <pa...@gmail.com>]

On Mon, Jun 22, 2009 at 1:51 PM, Jim McCoy<ji...@gmail.com> wrote:
> Something that would be nice to have for 1.0 or at least somewhere on
> the roadmap is full support for Erlang in views and in other bits of
> the codebase that accept functions as input.  I know that Javascript
> is quite popular with most of the Couch team, but it would be nice to
> know that Erlang could be used from top to bottom in the Couch stack
> rather than having to remember which bits are in Erlang and which are
> in Javascript (at least for those of us who are in that odd minority
> that do more Erlang coding than Javascript coding...)
>
>
> jim
>
> p.s.  I am not necessarily suggesting that this task be the focus of
> anyone on the dev team, but it would be nice to know that such an
> effort would be acceptable to the powers that be before someone spends
> time working on it...
>

It would even be encouraged!

Re: The 1.0 Thread

[Noah Slater <ns...@apache.org>]

On Mon, Jun 22, 2009 at 10:51:27AM -0700, Jim McCoy wrote:
> p.s.  I am not necessarily suggesting that this task be the focus of
> anyone on the dev team, but it would be nice to know that such an
> effort would be acceptable to the powers that be before someone spends
> time working on it...

You are the powers that be!

-- 
Noah Slater, http://tumbolia.org/nslater

Re: The 1.0 Thread

[Mark Hammond <sk...@gmail.com>]

On 23/06/2009 3:51 AM, Jim McCoy wrote:
> Something that would be nice to have for 1.0 or at least somewhere on
> the roadmap is full support for Erlang in views and in other bits of
> the codebase that accept functions as input.  I know that Javascript
> is quite popular with most of the Couch team, but it would be nice to
> know that Erlang could be used from top to bottom in the Couch stack
> rather than having to remember which bits are in Erlang and which are
> in Javascript (at least for those of us who are in that odd minority
> that do more Erlang coding than Javascript coding...)

Please check out https://issues.apache.org/jira/browse/COUCHDB-377 and 
feel free to update as per Damien's recent comments - I'm very very new 
to erlang and very busy at the moment, so it might take me a while...

Cheers,

Mark

Re: The 1.0 Thread

[Jim McCoy <ji...@gmail.com>]

Something that would be nice to have for 1.0 or at least somewhere on
the roadmap is full support for Erlang in views and in other bits of
the codebase that accept functions as input.  I know that Javascript
is quite popular with most of the Couch team, but it would be nice to
know that Erlang could be used from top to bottom in the Couch stack
rather than having to remember which bits are in Erlang and which are
in Javascript (at least for those of us who are in that odd minority
that do more Erlang coding than Javascript coding...)


jim

p.s.  I am not necessarily suggesting that this task be the focus of
anyone on the dev team, but it would be nice to know that such an
effort would be acceptable to the powers that be before someone spends
time working on it...

Re: The 1.0 Thread

[Scott Shumaker <ss...@gmail.com>]

Some of our views are invoked with stale=ok - but most of them can't
afford to be - if a user takes an action, typically they need to see
the results of that action if they continue on the site.

We're using CouchDB as our main persistent storage underlying our
entire site - which has data that is pretty darn complex.  We have
very little back-end code - we do almost everything in run-time on the
client in Javascript - loading in complex object hierarchies and
resolving cross-object pointers.  So we use views very heavily,
because typically you're loading a lot of different objects at once.

We are using the view filter patch, which helped a tiny bit - but our
data set isn't even that large yet (40k objects, admittedly many are
several K in size, or more) and the performance isn't great.  So I'm
definitely very concerned.  If we had the time to learn Erlang and the
CouchDB codebase, maybe we'd muck around to try to find optimizations.
 In lieu of that, we have to keep our fingers crossed for some
speedups, or failing that abandon CouchDB entirely in the next few
months.  :(

Scott


On Tue, Jun 30, 2009 at 11:52 AM, Peter Hsu<ph...@abaca.com> wrote:
> Scott - I don't know the details of your application.  What we tend to do with our couch-backed app is rely on stale=ok for user-facing views.  We then run a cron to update the view so that users will never get really stale data.  This is acceptable for our application, and may be for yours?
>
> +1 on view performance
> +1 on erlang view server
> +1 on consolidating multiple views

RE: The 1.0 Thread

[Peter Hsu <ph...@abaca.com>]

Scott - I don't know the details of your application.  What we tend to do with our couch-backed app is rely on stale=ok for user-facing views.  We then run a cron to update the view so that users will never get really stale data.  This is acceptable for our application, and may be for yours?

+1 on view performance
+1 on erlang view server
+1 on consolidating multiple views

-----Original Message-----
From: Scott Shumaker [mailto:sshumaker@gmail.com] 
Sent: Tuesday, June 30, 2009 11:29 AM
To: dev@couchdb.apache.org
Subject: Re: The 1.0 Thread

+3 on view performance (our team).  :)

It's a real problem for our application (currently in private alpha).
Right now, we're doing what we can to speed things up - we only have
40k documents in the database, but since so many of the views are user
facing, we find that views  often time out on the client side, causing
a broken webpage.  And adding a very simple new view takes on the
order of 8-10 minutes to run.  And to make matters worse, when there's
one new view, all the other views in the system (which should be
mostly up to date) bog down, as if they're waiting for the new view to
finish.  So it means every time we deploy we have to take the site
down for a while.  Last week a  push forced us to bring the site down
for 2 hours while couch reindexed.

We're in the middle of moving documents to different databases to try
to mitigate this problem, but the view performance is so awful that I
can only imagine there's some pretty low hanging fruit in terms of
performance.  As a comparison, we have a custom search server that
pulls docs from couchdb (using all_docs_by_seq), generating internal
searchable structures, and it is way faster to start up than it takes
to index a new view.

On Sat, Jun 27, 2009 at 10:01 PM, Benoit Chesneau<bc...@gmail.com> wrote:
> I'm agree with all the list aboive. It would be nice to have a way to
> combine views too :
>
> - multiples views results in one
> - possibility to map a view results and eventually reduce it too
> - chain both.
>
> Without it it require a lot of DIY and lot of calls if you do it via
> http which is sometimes required.
>
> -benoit
>

Re: The 1.0 Thread

[Scott Shumaker <ss...@gmail.com>]

+3 on view performance (our team).  :)

It's a real problem for our application (currently in private alpha).
Right now, we're doing what we can to speed things up - we only have
40k documents in the database, but since so many of the views are user
facing, we find that views  often time out on the client side, causing
a broken webpage.  And adding a very simple new view takes on the
order of 8-10 minutes to run.  And to make matters worse, when there's
one new view, all the other views in the system (which should be
mostly up to date) bog down, as if they're waiting for the new view to
finish.  So it means every time we deploy we have to take the site
down for a while.  Last week a  push forced us to bring the site down
for 2 hours while couch reindexed.

We're in the middle of moving documents to different databases to try
to mitigate this problem, but the view performance is so awful that I
can only imagine there's some pretty low hanging fruit in terms of
performance.  As a comparison, we have a custom search server that
pulls docs from couchdb (using all_docs_by_seq), generating internal
searchable structures, and it is way faster to start up than it takes
to index a new view.

On Sat, Jun 27, 2009 at 10:01 PM, Benoit Chesneau<bc...@gmail.com> wrote:
> I'm agree with all the list aboive. It would be nice to have a way to
> combine views too :
>
> - multiples views results in one
> - possibility to map a view results and eventually reduce it too
> - chain both.
>
> Without it it require a lot of DIY and lot of calls if you do it via
> http which is sometimes required.
>
> -benoit
>

Re: The 1.0 Thread

[Benoit Chesneau <bc...@gmail.com>]

I'm agree with all the list aboive. It would be nice to have a way to
combine views too :

- multiples views results in one
- possibility to map a view results and eventually reduce it too
- chain both.

Without it it require a lot of DIY and lot of calls if you do it via
http which is sometimes required.

-benoit

Re: The 1.0 Thread

[Chris Anderson <jc...@apache.org>]

On Thu, Jun 18, 2009 at 2:34 PM, Damien Katz<da...@apache.org> wrote:
> Okay, time to ask the question, what features do we need to get to 1.0?
>
> I'm going to list my must haves, and my nice to haves.
>
> Must have:
> - Document integrity checking: Using some sort of hashing scheme for end to
> end integrity checking of documents and attachments. Reusing the revision ID
> as the hash of the document might work, and has the benefit of allowing
> writing the same changes to 2 different servers and not causing a conflict.
> Also multiple clients can write the same change to a document and not get
> unnecessary conflicts.
> - Reader/Writer access databases and servers: Allow/disallow anonymous,
> users, groups.
> - Continuous replication: Keeping a constant connection and being able to
> replicate changes as soon as they happen.

Agree on the above listed ones.

> - Better testing: We need really some performance and stress testing as part
> of the source. And we need much better code coverage in general with the
> testing.

I think better testing can be a nice-to-have. I think we're on the
right track here and I don't ever feel held back by lack of tests. A
little Makefile integration so we have a more complete pre-commit
ritual would be the easiest way to add value here.

>
>
> Nice to have:
> - Hashing/CRC everything written to disk, data, metadata, index structures,
> etc. But optional, since many filesystems actively integrity-check disk
> data.

I think the document integrity checks will handle the part that really
matters, and the rest of this should fall on the OS. I'm not saying
CouchDB shouldn't eventually pick this up, but I'm comfortable having
this be post-1.0, especially as it doesn't effect the API.

> - Better full text integration: Out of the box integration and the ability
> intersect results with views, for easier result formatting. Lucene would be
> the primary FT engine, but we make it pluggable, much like the view engines
> are.

This would be nice but I think we need someone to really sponsor the
development. There are a lot of people using existing tools, maybe
some of them should start a thread discussing the good and bad parts
of their experience.

> - Attachment level replication: By tracking the revision when an attachment
> was modified, the replicator can avoid copying unchanged attachments to the
> target. The same can apply to json fields, but it's much less of a win
> there.

Is this going to fall out of deterministic revs already (document
integrity checking)?

> - Partitioning/sharding support: Ideally would be nice to have something
> that "just works" without a lot of setup.

I think CouchDB-Lounge 100% fills this void for now. Maybe we should
do more to link it to the project, even if that's just documentation.
Down the line we'll want to port CouchDB Lounge to Erlang and
integrate it, or perhaps even better would be to use Cliff Moon's
Dynomite project, but I see these as post-1.0 since users who need
medium sized clusters can get that now with Lounge, and large clusters
will take longer than we want to wait for 1.0 to implement and
production test.

> - Built-in authentication: A plug-in that authenticates HTTP users and
> assign them roles. It would use a couch database as a directory that
> contains users documents, etc.

I'd like to see this too, not sure if it should be a 1.0 blocker.
Depends on what others say.


> - Selective replication: The ability to replicate a subset of documents,
> using a javascript function as a selector.
> - Server side doc processing: The ability to POST data and have arbitrary
> server-side processing. The simplest case is posting a document to a Js
> handler that can do some data cleanup and add default values the document
> before saving it. But ideally would be able to interact with the full
> database

Personally I'd bump these two to the first list, but that's not really
an issue as I have a feeling they are gonna happen regardless. The
selective replication is a little easier as the API is pretty clear,
the server side update handler is more complex (especially if it
includes interacting with the full database), but either way I'm
motivated to help these patches along, and we're pretty close.

> - Scheduled replication: The ability to schedule replication every so often,
> like a cron job. But this can be done with an actual cron job and CURL, so
> it's not critical to have it built-in.

Yawn. If CouchDB is gonna grow it's own cron, that should come after 1.0 imho.


>
> And anyone who wants to take on any of these issues: mine, yours or anyone
> else's, just do it. Read code, mail dev@ with questions and advice, write
> some code, repeat.
>

Thanks for putting the list together, Damien. I agree about Nathan's
Windows requests, but I'm not a user.

The only other thing that jumps out at me is:

- An application (design doc) URL rewrite handler. We've discussed
this on other threads, I think the ideas are there we just need to
refine an implementation. I'd be fine seeing this come after 1.0, but
if it's ready it should go in.

Chris



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Re: The 1.0 Thread

[Brian Candler <B....@pobox.com>]

+1 for anything which can improve performance of generating views,
especially when importing large datasets.

Maybe it's just me, but CouchDB "feels" much slower than an RDBMS for this.
That is, an RDBMS might limit you to only a couple of hundred inserts per
second, but at least all your indexes are being built too. With CouchDB I
can bulk-insert 600+ docs per second on an old laptop but the subsequent
view generation takes far, far longer.

CouchDB ought to have some advantages over the RDBMS: all the 'new' docs are
contiguous on disk, and it can add to its btrees in bulk, rather than once
per insert. However it has handicaps too, notably
erlang->JSON->JS->JSON->erlang serialisation and the associated context
switches. I only just started profiling this and I don't see anything
obviously bad, but I have a real-world app which only manages to index 10 or
20 docs per sec (emitting an average of ~10 K/V pairs per doc), admittedly
with a bit of regexp munging in the Javascript view code.

Re: The 1.0 Thread

[Antony Blakey <an...@gmail.com>]

On 22/06/2009, at 7:26 AM, Paul Davis wrote:

> Also +lots on deterministic revisions. As a side note, we've been
> worrying a bit about how to calculate the revision id's in the face of
> JSON differences in clients. I make a motion that we just stop caring
> and define how we calculate the signature. Ie, instead of calling it
> canonical JSON just call it, "The CouchDB deterministic revision
> algorithm" or some such. Then we can feel free to punt on any of the
> awesome issues we'd run into with a full canonical JSON standard.

I haven't seen the recent discussions about canonicalisation, but IMO  
a minimum requirement is that the infoset <-> serialisation mapping  
must be 1:1, which requires completeness and prescription. Doing  
unicode normalisation (NFC probably) is IMO also an absolute  
requirement - it's virtually impossible to construct documents by hand  
with repeatable results without it.

Unicode normalisation is an issue for clients because it requires they  
have access to a Unicode NFC function.

Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

It is as useless to argue with those who have renounced the use of  
reason as to administer medication to the dead.
   -- Thomas Jefferson

Re: The 1.0 Thread

[Paul Davis <pa...@gmail.com>]

On Sat, Jun 20, 2009 at 5:24 PM, Robert Newson<ro...@gmail.com> wrote:
> If there's anything I could do to more tightly integrate
> couchdb-lucene with core to accomplish one of your bullet points, I'm
> listening. I pushed a 0.3 release last week and I'm pretty happy with
> the API, I don't see any major breaking changes coming any time soon,
> just new features as I get time.
>
> +1 million for "hash as revision". It would be very cool to be able to
> write the same change to N boxes and have that not be a conflict.
>
> B.
>
> On Fri, Jun 19, 2009 at 9:56 PM, Patrick Aljord<pa...@gmail.com> wrote:
>> +1 on ability to intersect (or union) multiple view results.
>> URL rewrite handler and Built-in authentication would be nice for couchapps.
>>
>

Also +lots on deterministic revisions. As a side note, we've been
worrying a bit about how to calculate the revision id's in the face of
JSON differences in clients. I make a motion that we just stop caring
and define how we calculate the signature. Ie, instead of calling it
canonical JSON just call it, "The CouchDB deterministic revision
algorithm" or some such. Then we can feel free to punt on any of the
awesome issues we'd run into with a full canonical JSON standard.

Paul Davis

Re: The 1.0 Thread

[Robert Newson <ro...@gmail.com>]

If there's anything I could do to more tightly integrate
couchdb-lucene with core to accomplish one of your bullet points, I'm
listening. I pushed a 0.3 release last week and I'm pretty happy with
the API, I don't see any major breaking changes coming any time soon,
just new features as I get time.

+1 million for "hash as revision". It would be very cool to be able to
write the same change to N boxes and have that not be a conflict.

B.

On Fri, Jun 19, 2009 at 9:56 PM, Patrick Aljord<pa...@gmail.com> wrote:
> +1 on ability to intersect (or union) multiple view results.
> URL rewrite handler and Built-in authentication would be nice for couchapps.
>

Re: The 1.0 Thread

[Patrick Aljord <pa...@gmail.com>]

+1 on ability to intersect (or union) multiple view results.
URL rewrite handler and Built-in authentication would be nice for couchapps.

Re: The 1.0 Thread

[Barry Wark <ba...@gmail.com>]

On Thu, Jun 18, 2009 at 2:34 PM, Damien Katz<da...@apache.org> wrote:
> Okay, time to ask the question, what features do we need to get to 1.0?
>
> I'm going to list my must haves, and my nice to haves.
>
> Must have:
> - Document integrity checking: Using some sort of hashing scheme for end to
> end integrity checking of documents and attachments. Reusing the revision ID
> as the hash of the document might work, and has the benefit of allowing
> writing the same changes to 2 different servers and not causing a conflict.
> Also multiple clients can write the same change to a document and not get
> unnecessary conflicts.
> - Reader/Writer access databases and servers: Allow/disallow anonymous,
> users, groups.
> - Continuous replication: Keeping a constant connection and being able to
> replicate changes as soon as they happen.
> - Better testing: We need really some performance and stress testing as part
> of the source. And we need much better code coverage in general with the
> testing.
>
>
> Nice to have:
> - Hashing/CRC everything written to disk, data, metadata, index structures,
> etc. But optional, since many filesystems actively integrity-check disk
> data.
> - Better full text integration: Out of the box integration and the ability
> intersect results with views, for easier result formatting. Lucene would be
> the primary FT engine, but we make it pluggable, much like the view engines
> are.

If I may, I would like to put a +1 on ability to intersect (or union)
multiple view results. This is the feature that's preventing
whole-hearted adoption of CouchDB for several applications at my
company. Lucene is close to a solution, but we really need proper
numeric comparisons, not just text comparisons. I don't think we can
commit any resources to making a patch happen on this front until
Jan/Feb. 2010. At that time, we would be willing to help make  this
happen as we're quite excited about CouchDB but are being held up by
the lack of boolean view combinations (the data sets in question are
too large to handle the logic client-side).


-Barry

> - Attachment level replication: By tracking the revision when an attachment
> was modified, the replicator can avoid copying unchanged attachments to the
> target. The same can apply to json fields, but it's much less of a win
> there.
> - Partitioning/sharding support: Ideally would be nice to have something
> that "just works" without a lot of setup.
> - Built-in authentication: A plug-in that authenticates HTTP users and
> assign them roles. It would use a couch database as a directory that
> contains users documents, etc.
> - Selective replication: The ability to replicate a subset of documents,
> using a javascript function as a selector.
> - Server side doc processing: The ability to POST data and have arbitrary
> server-side processing. The simplest case is posting a document to a Js
> handler that can do some data cleanup and add default values the document
> before saving it. But ideally would be able to interact with the full
> database
> - Scheduled replication: The ability to schedule replication every so often,
> like a cron job. But this can be done with an actual cron job and CURL, so
> it's not critical to have it built-in.
>
> There are probably a bunch of things I forgot about.
>
> Respond to this with your must haves and nice to haves. No promises you'll
> get your way (no guarantee for me for that matter), but lets start talking
> about it.
>
> And anyone who wants to take on any of these issues: mine, yours or anyone
> else's, just do it. Read code, mail dev@ with questions and advice, write
> some code, repeat.
>
> -Damien
>