You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Wout Mertens <wm...@cisco.com> on 2009/01/27 14:38:06 UTC
Current CouchDB state?
Hi there,
I'm a big fan of CouchDB and I want to use it in production but I'm
holding out for a somewhat stable release. In preparation however I
would like to know where CouchDB stands in High Availability and/or
Scalability.
I know that Erlang promises that you can run a single program across
multiple CPUs and even systems at the same time, and you can upgrade
the program while it is handling requests. I presume Erlang also
handles the case where a system goes offline?
Furthermore, I know that the view server runs separately from the
CouchDB process, and it can in principle split the map and reduce
phases across many compute nodes.
Finally, I know that the replication works. Is there a "standard" way
of using it with supporting programs?
So my question is: How much of this is already a reality? Is any
testing being done for these scenarios? Is this on the roadmap for 1.0?
Thanks,
Wout.
Re: Current CouchDB state?
Posted by Jan Lehnardt <ja...@apache.org>.
On 28 Jan 2009, at 12:55, Brian Candler wrote:
> On Wed, Jan 28, 2009 at 11:40:18AM +0100, Jan Lehnardt wrote:
>>> BTW, I do think the atomic nature of bulk_docs is useful and
>>> should be
>>> kept,
>>> as it's the only way to get "transaction" semantics at the moment.
>>
>> We won't be able to guarantee transactions in a multi-node setup.
>
> This is true - although I presume you mean that the transaction can
> succeed
> on one node, but will then cause a conflict upon replication. If so,
> I'd say
> this is the case with single document updates too (but maybe I have
> misunderstood).
This refers to partitioned databases, where a single database lives on
multiple nodes. A transaction would be an effective N-phase commit
(where N is the number of nodes) and I hope we all agree that N-phase
commit is not a good option for us (heck, even banks don't use it
anymore
these days.
Cheers
Jan
--
>
>
> Replication conflict resolution is something which I don't have a
> handle on
> yet, I'm afraid, and doesn't seem to be well documented. (*)
>
> Also: is there some guidance as to how to write applications in the
> absence
> of transactions? Take traditional double-entry accounts as an example.
> Should we create a single document for each transaction, and use a
> view to
> get the balance of each account? e.g.
>
> {"amount": ...
> "credit_ac": [account_name, accounting_period]
> "debit_ac": [account_name, accounting_period]}
>
> So a view will give you [account_name, accounting_period] => balance
>
> Replication conflict resolution is avoided simply on the basis that
> documents never change, they are only created.
>
> Anyway, have I got the right sort of idea here? IANAA, and I don't
> think
> this system looks very "failsafe", in the sense that if a
> transaction were
> to be lost for any reason, the whole system would still remain in
> balance so
> it would be very hard to detect.
>
> Regards,
>
> Brian.
>
> (*) I can see that replication can result in a document with conflicts
> marked; but how is it resolved? Is the conflict marked as resolved
> as soon
> as any client reads the document and writes a new revision?
>
> If so, this means that every time you make an update to a document,
> you must
> first check to see if it is in conflict, and resolve the conflict
> before
> proceeding. Is that correct?
>
Transactional CouchDB (was Re: Current CouchDB state?)
Posted by Wout Mertens <wm...@cisco.com>.
Replying to snippet:
On Jan 28, 2009, at 12:55 PM, Brian Candler wrote:
> Also: is there some guidance as to how to write applications in the
> absence
> of transactions? Take traditional double-entry accounts as an example.
> Should we create a single document for each transaction, and use a
> view to
> get the balance of each account? e.g.
>
> {"amount": ...
> "credit_ac": [account_name, accounting_period]
> "debit_ac": [account_name, accounting_period]}
>
> So a view will give you [account_name, accounting_period] => balance
>
> Replication conflict resolution is avoided simply on the basis that
> documents never change, they are only created.
>
> Anyway, have I got the right sort of idea here? IANAA, and I don't
> think
> this system looks very "failsafe", in the sense that if a
> transaction were
> to be lost for any reason, the whole system would still remain in
> balance so
> it would be very hard to detect.
It seems to me that to properly implement double-entry accounts, you
would in fact use two separate databases. Note however that double-
entry was invented to detect human mistakes. With computers you would
probably want to detect inconsistencies arising from computer
problems; this is probably different from what humans produce.
So in this case you probably want to write somewhere in a very trusted
system "I'm about to perform this transaction", then perform the
transaction (two records, debit and credit, one for each account?) and
finally put in the trusted system "I did the transaction".
A lot like log-structured filesystems, in fact. Of course, I never
coded anything like that so there are probably better ways.
Wout.
Re: Current CouchDB state?
Posted by Brian Candler <B....@pobox.com>.
On Wed, Jan 28, 2009 at 11:40:18AM +0100, Jan Lehnardt wrote:
>> BTW, I do think the atomic nature of bulk_docs is useful and should be
>> kept,
>> as it's the only way to get "transaction" semantics at the moment.
>
> We won't be able to guarantee transactions in a multi-node setup.
This is true - although I presume you mean that the transaction can succeed
on one node, but will then cause a conflict upon replication. If so, I'd say
this is the case with single document updates too (but maybe I have
misunderstood).
Replication conflict resolution is something which I don't have a handle on
yet, I'm afraid, and doesn't seem to be well documented. (*)
Also: is there some guidance as to how to write applications in the absence
of transactions? Take traditional double-entry accounts as an example.
Should we create a single document for each transaction, and use a view to
get the balance of each account? e.g.
{"amount": ...
"credit_ac": [account_name, accounting_period]
"debit_ac": [account_name, accounting_period]}
So a view will give you [account_name, accounting_period] => balance
Replication conflict resolution is avoided simply on the basis that
documents never change, they are only created.
Anyway, have I got the right sort of idea here? IANAA, and I don't think
this system looks very "failsafe", in the sense that if a transaction were
to be lost for any reason, the whole system would still remain in balance so
it would be very hard to detect.
Regards,
Brian.
(*) I can see that replication can result in a document with conflicts
marked; but how is it resolved? Is the conflict marked as resolved as soon
as any client reads the document and writes a new revision?
If so, this means that every time you make an update to a document, you must
first check to see if it is in conflict, and resolve the conflict before
proceeding. Is that correct?
Re: Current CouchDB state?
Posted by Chris Anderson <jc...@apache.org>.
On Wed, Jan 28, 2009 at 3:03 AM, Noah Slater <ns...@apache.org> wrote:
> On Wed, Jan 28, 2009 at 09:30:19PM +1030, Antony Blakey wrote:
>>
>> On 28/01/2009, at 9:10 PM, Jan Lehnardt wrote:
>>
>>>
>>> On 28 Jan 2009, at 11:31, Brian Candler wrote:
>>>
>>>> BTW, I do think the atomic nature of bulk_docs is useful and should
>>>> be kept,
>>>> as it's the only way to get "transaction" semantics at the moment.
>>>
>>> We won't be able to guarantee transactions in a multi-node setup.
>>
>> And there's a universe of single-node applications.
>
> I would prefer a predictable interface over single-node special-casing.
>
We had a book comment mentioning the usefulness of giving
application-level control over whether to run a particular computation
on a cluster, or on a single node, for the purpose of taking advantage
of the different side of the CAP theorem. It made me rethink the idea
that we should eschew single-node transactions.
For instance, an application could replicate a set of docs to a single
node, run a transaction across them, and then replicate back. Or you
might put some types of data on a big fast machine, but spread most
data around commodity cluster machines.
It's worth thinking about - there are things that get much harder to
do, without multi-doc transactions...
Chris
--
Chris Anderson
http://jchris.mfdz.com
Re: Current CouchDB state?
Posted by Noah Slater <ns...@apache.org>.
On Wed, Jan 28, 2009 at 08:52:55AM -0500, Jim Jagielski wrote:
> Yes, and it's called the ASF process.
Yes, so if you want to be a PMC member, there is provision for that.
> All of the above seems to reinforce the concept that one person's vote
> is more important than any else's on the PMC. I am struggling to
> understand the comprehension failure here. This is *not* the way
> the ASF works.
How could you possibly understand that from what I wrote?
As a PMC we are generally happy to be guided by Damien, and when we disagree
with him we do so as any normal member of the PMC. We have votes, we vote how we
feel, even if that means disagreement. If Damien suggested something that I
thought was totally bunk, I would happily call it as much on the mailing list.
> The ASF has for a long, long, long time avoided and actively restricted
> such concepts as "key developer" or "main author" or other such things.
> The idea is to create a community around the code that does not depend
> on a single person, nor is revolved around a single person. It's to
> create a community where people can make significant contributions
> and know that their vote counts just as much as anyones. It's to create
> a community that *survives*.
Okay, this is pushing into territory that I'm not comfortable traversing. The
PMC has never had a private vote where we all formally agreed to listen to
Damien and his One True Vision. I can only speak for my self here.
What are you suggesting I do, as a PMC member? Are you suggesting that I abandon
my respect for Damien and his existing vision? Are you asking me to not seek his
feedback and guidance on design issues?
At no point in any of this have I said that Damien is a true BDFL in this sense,
sorry if my language was confusing. A case in point is this stupid newlines patch
that I'm trying to get resolved. Damien doesn't like it, but that gives him no
special privileges. I'm not going to back down in fear of upsetting him.
I am absolutely not, in anyway, saying that Damien has veto power over anything
or that he is organisationally more important than the rest of the PMC. I am, on
the other hand, saying that he is a socially respected member of the community
and that the PMC tends to look to him for guidance on technical issues. Guidance
doesn't mean unequivocal acceptance. I don't understand the confusion, and I
don't know how many times I can reword this concept.
--
Noah Slater, http://tumbolia.org/nslater
Re: Current CouchDB state?
Posted by Chris Anderson <jc...@apache.org>.
On Wed, Jan 28, 2009 at 5:52 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> On Jan 28, 2009, at 8:23 AM, Noah Slater wrote:
>
>> On Wed, Jan 28, 2009 at 08:03:17AM -0500, Jim Jagielski wrote:
>>>
>>> Sorry, that's not how the ASF works. There is no concept of BDFL.
>>> Everyone's vote and opinion on the PMC has the exact same weight.
>>> Anyone can legally and officially veto anything (with a solid technical
>>> reason).
>>>
>>> If the concept of BDFL is crucial to CouchDB, then it does not
>>> belong under the ASF...
>>
>> ...
>>>
>>> As long as it *is* the vision of the PMC, and not the vision of one
>>> person and the PMC, like sheep, simply accept it...
>>
>> I am really struggling to understand the comprehension failure here.
>>
>> In summary:
>>
>> * The ASF works by giving every member of the PMC a binding vote in all
>> decisions. Each member of the PMC has the same power.
>>
>> * By definition, each member of the PMC has the privilege to choose how
>> they
>> vote and for what reasons.
>>
>> * If the members of the PMC decide on a personal basis that they wish to
>> accept guidance from one person as a BDFL type figure that is absolutely
>> their imperative.
>>
>> * You can see from previous votes that we are all happy to disagree with
>> each
>> other, including Damien, when we feel a particular way about something.
>> I
>> resent your implication that there is any "sheepishness" about the
>> process.
>>
>> * To suggest any alternative to this is to be encouraging us as a PMC to
>> purposefully disagree with each other and cause problems simply because
>> we
>> have the power to do so.
>>
>> * To imply that we *mindlessly* follow Damien about like lost sheep is
>> both
>> insulting and incredibly presumptuous.
>>
>> As a PMC member I am happy to be guided by Damien's vision for CouchDB.
>>
>> If you wish to be a PMC member who isn't, there is a process in place for
>> that.
>
> Yes, and it's called the ASF process.
>
> All of the above seems to reinforce the concept that one person's vote
> is more important than any else's on the PMC. I am struggling to
> understand the comprehension failure here. This is *not* the way
> the ASF works.
>
I know this has been covered in subsequent replies, but I was doing
that whole sleep thing while this thread got exciting.
I think CouchDB fits well into the Apache Way. When I disagree with
Damien or any of the other PMCs, I don't hesitate to make it known. On
the other hand, there are parts of the vision and the code, that I
trust Damien to understand better than I do. At the same time, there
are aspects of CouchDB's vision and code where I very much trust my
own instincts and experience. I think that each of the PMC members
feels the same way, and that shows up, in discussions and the commit
log, as each of us have parts of the project we focus most on.
The governance questions on this thread are interesting. Nothing like
first-hand experience to show the value of an institution like the
ASF.
The transactional nature of bulk_docs has not been decided. In my
opinion, nothing is decided until it hits code. I think we could have
a fruitful technical discussion on this topic, but the accusatory tone
with which it was begun may not be the best strategy.
> It's to create
> a community that *survives*.
I've been putting in long days working in the guts of CouchDB, and
querying Damien when I get stuck, even when some problems might be
more quickly handled by him, precisely because having a single person
know the whys and hows of CouchDB's "deep magic" is not a sustainable
strategy. It will take time, but I have every hope that this project
will outgrow its original roots, and that we'll have a whole group of
developers who are capable of making the tough technical decisions.
I'm not in any way worried about the community. I think this vigorous
discussion is a sign of a healthy group of interested people. It's too
bad that it started from such a strange (and in my opinion, unfounded)
perspective.
The CouchDB motto is "Relax". This means if someone makes an offhand
comment about a potential API change, which seems like a bad idea to
you, it's best to reply about the API change, not about the manner in
which you first heard of it's possibility. I don't think it will do
the project any good to have everyone stepping on tip-toes to avoid
giving the wrong impression about what's been "decided".
Chris
(I hope I don't come off too grumpy here - running on 1/2 my usual sleep today.)
--
Chris Anderson
http://jchris.mfdz.com
Re: Current CouchDB state?
Posted by Jan Lehnardt <ja...@apache.org>.
Hi Jim,
thanks for speaking up and raising concerns as well.
On 28 Jan 2009, at 14:52, Jim Jagielski wrote:
>
> On Jan 28, 2009, at 8:23 AM, Noah Slater wrote:
>
>> On Wed, Jan 28, 2009 at 08:03:17AM -0500, Jim Jagielski wrote:
>>>
>> If you wish to be a PMC member who isn't, there is a process in
>> place for that.
>
> Yes, and it's called the ASF process.
>
> All of the above seems to reinforce the concept that one person's vote
> is more important than any else's on the PMC. I am struggling to
> understand the comprehension failure here. This is *not* the way
> the ASF works.
The idea of the BDFL (at least for us) is not to have a more important
vote but to help the PMC decide issues that are hard to decide or need
wise judgement. In the case of a distributed "Notes-style" database,
Damien is clearly the expert among us.
Besides, we didn't even officially "appoint" Damien and new committers
or PMC members do not have to subscribe to the current PMC's ideas.
> The ASF has for a long, long, long time avoided and actively
> restricted
> such concepts as "key developer" or "main author" or other such
> things.
> The idea is to create a community around the code that does not depend
> on a single person, nor is revolved around a single person. It's to
> create a community where people can make significant contributions
> and know that their vote counts just as much as anyones. It's to
> create
> a community that *survives*.
We're working on getting Damien out of the place of being the lead-dev,
but for the time being, he has the clearest ideas of what needs to be
done
and we happen to agree with his judgement. He's also the only one being
paid to work on lower CouchDB internals.
I think what binds us is the "Technical Overview"[1] document that
resides
on http://couchdb.apache.org/ It admittedly could do with some updating
but it lays out the general idea of CouchDB and a lot of the specifics
and
that is what so far everybody who joined the community agrees on.
For the future, it is the PMC's job to refine the document and make sure
it aligns with development goals and community and developer requests.
Cheers
Jan
--
[1]: http://couchdb.apache.org/docs/overview.html
Re: Current CouchDB state?
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Jan 28, 2009, at 8:23 AM, Noah Slater wrote:
> On Wed, Jan 28, 2009 at 08:03:17AM -0500, Jim Jagielski wrote:
>> Sorry, that's not how the ASF works. There is no concept of BDFL.
>> Everyone's vote and opinion on the PMC has the exact same weight.
>> Anyone can legally and officially veto anything (with a solid
>> technical
>> reason).
>>
>> If the concept of BDFL is crucial to CouchDB, then it does not
>> belong under the ASF...
> ...
>> As long as it *is* the vision of the PMC, and not the vision of one
>> person and the PMC, like sheep, simply accept it...
>
> I am really struggling to understand the comprehension failure here.
>
> In summary:
>
> * The ASF works by giving every member of the PMC a binding vote in
> all
> decisions. Each member of the PMC has the same power.
>
> * By definition, each member of the PMC has the privilege to choose
> how they
> vote and for what reasons.
>
> * If the members of the PMC decide on a personal basis that they
> wish to
> accept guidance from one person as a BDFL type figure that is
> absolutely
> their imperative.
>
> * You can see from previous votes that we are all happy to disagree
> with each
> other, including Damien, when we feel a particular way about
> something. I
> resent your implication that there is any "sheepishness" about
> the process.
>
> * To suggest any alternative to this is to be encouraging us as a
> PMC to
> purposefully disagree with each other and cause problems simply
> because we
> have the power to do so.
>
> * To imply that we *mindlessly* follow Damien about like lost sheep
> is both
> insulting and incredibly presumptuous.
>
> As a PMC member I am happy to be guided by Damien's vision for
> CouchDB.
>
> If you wish to be a PMC member who isn't, there is a process in
> place for that.
Yes, and it's called the ASF process.
All of the above seems to reinforce the concept that one person's vote
is more important than any else's on the PMC. I am struggling to
understand the comprehension failure here. This is *not* the way
the ASF works.
The ASF has for a long, long, long time avoided and actively restricted
such concepts as "key developer" or "main author" or other such things.
The idea is to create a community around the code that does not depend
on a single person, nor is revolved around a single person. It's to
create a community where people can make significant contributions
and know that their vote counts just as much as anyones. It's to create
a community that *survives*.
Re: Current CouchDB state?
Posted by Noah Slater <ns...@apache.org>.
On Wed, Jan 28, 2009 at 08:03:17AM -0500, Jim Jagielski wrote:
> Sorry, that's not how the ASF works. There is no concept of BDFL.
> Everyone's vote and opinion on the PMC has the exact same weight.
> Anyone can legally and officially veto anything (with a solid technical
> reason).
>
> If the concept of BDFL is crucial to CouchDB, then it does not
> belong under the ASF...
...
> As long as it *is* the vision of the PMC, and not the vision of one
> person and the PMC, like sheep, simply accept it...
I am really struggling to understand the comprehension failure here.
In summary:
* The ASF works by giving every member of the PMC a binding vote in all
decisions. Each member of the PMC has the same power.
* By definition, each member of the PMC has the privilege to choose how they
vote and for what reasons.
* If the members of the PMC decide on a personal basis that they wish to
accept guidance from one person as a BDFL type figure that is absolutely
their imperative.
* You can see from previous votes that we are all happy to disagree with each
other, including Damien, when we feel a particular way about something. I
resent your implication that there is any "sheepishness" about the process.
* To suggest any alternative to this is to be encouraging us as a PMC to
purposefully disagree with each other and cause problems simply because we
have the power to do so.
* To imply that we *mindlessly* follow Damien about like lost sheep is both
insulting and incredibly presumptuous.
As a PMC member I am happy to be guided by Damien's vision for CouchDB.
If you wish to be a PMC member who isn't, there is a process in place for that.
Thanks,
--
Noah Slater, http://tumbolia.org/nslater
Re: Current CouchDB state?
Posted by Jim Jagielski <ji...@jaguNET.com>.
As long as the PMC continues to operate as a PMC and to
encourage and grow the community around it, and as long as
they are aware of the risks and are avoiding them, then I
think this issue is resolved...
Re: Current CouchDB state?
Posted by Damien Katz <da...@apache.org>.
On Jan 28, 2009, at 8:59 AM, Jim Jagielski wrote:
>
> On Jan 28, 2009, at 8:26 AM, Kerr Rainey wrote:
>
>> 2009/1/28 Jim Jagielski <ji...@jagunet.com>:
>>> As long as it *is* the vision of the PMC, and not the vision of one
>>> person and the PMC, like sheep, simply accept it...
>>
>> I don't think the PMC are following anyone as sheep. A single person
>> had a vision, expended a phenomenal amount of effort and risk in
>> bringing that vision to life. He then gave it away. That
>> engenders a
>> huge amount respect and willingness to listen to there opinion of the
>> subject. I don't think it's a sign of weakness that the opinion of
>> the person who has done the most work and understands the code the
>> best is sought out by the rest of the PMC.
>>
>
> I'm not saying that one should not give that person's
> opinion a fair weighing. I'm also saying that vision
> is what the PMC defines, as a team, and as a group of
> people who all have a vision. That's what being on
> the PMC is about, being able to define that vision.
> Just just "rubber stamping" someone else's vision.
>
> I'm also not saying this is what is occurring; I'm only
> saying this a very real risk, esp for a recently graduated
> ASF TLP. If developers leave, or decide never to join
> because they fear that their voices will never be heard,
> or their input or "vision" will ever be valid, unless
> it happens to align with a single persons', then hopefully
> people see the problem and concern.
My vote counts the same as another else. If my opinion counts more
amongst the other PMC members and other members of the community, it
is not by decree or force. So perhaps we can agree that it's based on
merit, or at least the appearance of merit.
Developing a completely new distributed database platform is a lot of
work and takes a tremendous amount of mental effort. I and the other
project members try to consider the community at large with every
decision, but we simply cannot move forward if we need to address
everyones criticism and concerns for every decision. It's exhausting,
and no matter how much we try, we still can't make everyone happy. Yet
we must make progress.
The reason people keep deferring to my opinion here because they know
I have an opinion on this, one I have articulated to the other project
members and gotten consensus around. However, the other project
members cannot be expected to recall every detail of every decision
and why it was necessary. And we cannot be expected to answer in
detail every criticism and concern of "why isn't it another way?" And
so people defer to previously made decisions, knowing they we're made
thoughtfully and respectfully, even if we can't remember all the
details right now. It's not ideal, but it's reality. We are trying our
best to push the project forward.
-Damien
Re: Current CouchDB state?
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Jan 28, 2009, at 8:26 AM, Kerr Rainey wrote:
> 2009/1/28 Jim Jagielski <ji...@jagunet.com>:
>> As long as it *is* the vision of the PMC, and not the vision of one
>> person and the PMC, like sheep, simply accept it...
>
> I don't think the PMC are following anyone as sheep. A single person
> had a vision, expended a phenomenal amount of effort and risk in
> bringing that vision to life. He then gave it away. That engenders a
> huge amount respect and willingness to listen to there opinion of the
> subject. I don't think it's a sign of weakness that the opinion of
> the person who has done the most work and understands the code the
> best is sought out by the rest of the PMC.
>
I'm not saying that one should not give that person's
opinion a fair weighing. I'm also saying that vision
is what the PMC defines, as a team, and as a group of
people who all have a vision. That's what being on
the PMC is about, being able to define that vision.
Just just "rubber stamping" someone else's vision.
I'm also not saying this is what is occurring; I'm only
saying this a very real risk, esp for a recently graduated
ASF TLP. If developers leave, or decide never to join
because they fear that their voices will never be heard,
or their input or "vision" will ever be valid, unless
it happens to align with a single persons', then hopefully
people see the problem and concern.
Re: Current CouchDB state?
Posted by Kerr Rainey <ke...@gmail.com>.
2009/1/28 Jim Jagielski <ji...@jagunet.com>:
> As long as it *is* the vision of the PMC, and not the vision of one
> person and the PMC, like sheep, simply accept it...
I don't think the PMC are following anyone as sheep. A single person
had a vision, expended a phenomenal amount of effort and risk in
bringing that vision to life. He then gave it away. That engenders a
huge amount respect and willingness to listen to there opinion of the
subject. I don't think it's a sign of weakness that the opinion of
the person who has done the most work and understands the code the
best is sought out by the rest of the PMC.
--
Kerr
Re: Current CouchDB state?
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Wed, Jan 28, 2009 at 12:42:26PM +0000, Kerr Rainey wrote:
> 2009/1/28 Antony Blakey <an...@gmail.com>:
> >> The project wasn't set up as blank page with a group of people
> >> discussing what they wanted to do, it was a single persons vision. A
> >> vision that other people thought was cool and have come together to
> >> make it better.
> >
> > And now it's an Apache project, so it's no longer appropriate for it to be
> > one person's vision.
>
> Yes, I agree. But the project has momentum behind the original
> vision. A vision that others have picked up and run with as a
> community. This a good thing because a project without vision is
> doomed to fail.
>
> This doesn't mean that the direction can't be discussed or changed,
> but you need to convince people that are all running in one direction
> that they need to change course.
>
> You seem to get upset when decisions are made that fit well with the
> basic guiding vision that was there when the project was initiated. I
> don't think this is a single persons vision any more, rather the
> vision of the PMC and a large chunk of the rest of the community.
> Yes, there could probably be a better laid out roadmap with
> documentation and plans, bat as always that's at the bottom of the to
> do list.
>
As long as it *is* the vision of the PMC, and not the vision of one
person and the PMC, like sheep, simply accept it...
--
===========================================================================
Jim Jagielski [|] jim@jaguNET.com [|] http://www.jaguNET.com/
"Great is the guilt of an unnecessary war" ~ John Adams
Re: Current CouchDB state?
Posted by Noah Slater <ns...@apache.org>.
On Wed, Jan 28, 2009 at 12:42:26PM +0000, Kerr Rainey wrote:
> So if you have something you want to discuss, bring it up, but please
> leave the complaining that it's not been discussed *yet* at the door.
Agreed, and the last thing we need is stop energy.
--
Noah Slater, http://tumbolia.org/nslater
Re: Current CouchDB state?
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Wed, Jan 28, 2009 at 12:35:34PM +0000, Noah Slater wrote:
> On Wed, Jan 28, 2009 at 10:27:42PM +1030, Antony Blakey wrote:
> >> The project wasn't set up as blank page with a group of people
> >> discussing what they wanted to do, it was a single persons vision. A
> >> vision that other people thought was cool and have come together to
> >> make it better.
> >
> > And now it's an Apache project, so it's no longer appropriate for it to
> > be one person's vision.
>
> I call bullshit. As the PMC we are happily bound to the rules of the Apache
> Software Foundation, but if we as individuals choose to organise ourselves
> around Damien as the BDFL then that is our choice to do so.
>
Sorry, that's not how the ASF works. There is no concept of BDFL.
Everyone's vote and opinion on the PMC has the exact same weight.
Anyone can legally and officially veto anything (with a solid technical
reason).
If the concept of BDFL is crucial to CouchDB, then it does not
belong under the ASF...
--
===========================================================================
Jim Jagielski [|] jim@jaguNET.com [|] http://www.jaguNET.com/
"Great is the guilt of an unnecessary war" ~ John Adams
Re: Current CouchDB state?
Posted by Noah Slater <ns...@apache.org>.
On Wed, Jan 28, 2009 at 10:27:42PM +1030, Antony Blakey wrote:
>> The project wasn't set up as blank page with a group of people
>> discussing what they wanted to do, it was a single persons vision. A
>> vision that other people thought was cool and have come together to
>> make it better.
>
> And now it's an Apache project, so it's no longer appropriate for it to
> be one person's vision.
I call bullshit. As the PMC we are happily bound to the rules of the Apache
Software Foundation, but if we as individuals choose to organise ourselves
around Damien as the BDFL then that is our choice to do so.
If you want to pull in another direction that is fine, and healthy. Call for
changes on the mailing list, or become a committer and push through your own
enhancements. But don't make false accusations of malpractice.
--
Noah Slater, http://tumbolia.org/nslater
Re: Current CouchDB state?
Posted by Kerr Rainey <ke...@gmail.com>.
2009/1/28 Antony Blakey <an...@gmail.com>:
>> The project wasn't set up as blank page with a group of people
>> discussing what they wanted to do, it was a single persons vision. A
>> vision that other people thought was cool and have come together to
>> make it better.
>
> And now it's an Apache project, so it's no longer appropriate for it to be
> one person's vision.
Yes, I agree. But the project has momentum behind the original
vision. A vision that others have picked up and run with as a
community. This a good thing because a project without vision is
doomed to fail.
This doesn't mean that the direction can't be discussed or changed,
but you need to convince people that are all running in one direction
that they need to change course.
You seem to get upset when decisions are made that fit well with the
basic guiding vision that was there when the project was initiated. I
don't think this is a single persons vision any more, rather the
vision of the PMC and a large chunk of the rest of the community.
Yes, there could probably be a better laid out roadmap with
documentation and plans, bat as always that's at the bottom of the to
do list.
And how much of it needs to be discussed? How many people here want
to spend time discussing core principals of what makes CouchDB
CouchDB? Do you want to discuss CouchDB becoming relational? It's an
Apache project now, has that been discussed since then, or is it OK to
assume that a core part of the original vision is that CouchDB is not
relational?
So if you have something you want to discuss, bring it up, but please
leave the complaining that it's not been discussed *yet* at the door.
--
Kerr
Re: Current CouchDB state?
Posted by Noah Slater <ns...@apache.org>.
On Wed, Jan 28, 2009 at 10:53:59PM +1030, Antony Blakey wrote:
> _bulk_docs is the only transactional mechanism in CouchDB. Alarm bells
> went off when I saw that it was effectively decided to remove that
> behaviour. And even in a cluster setup (as opposed to a partition-
> tolerant distributed system), it is possible to provide transactional
> behaviour. Sure, it's some work, with it's own set of issue, but IMO the
> benefit is enormous, and virtually impossible to do outside the system
> unless it provides something like an XA interface, which seems wrong.
Patches welcome. Aggressive complaints, less so.
--
Noah Slater, http://tumbolia.org/nslater
Re: Current CouchDB state?
Posted by Jan Lehnardt <ja...@apache.org>.
On 28 Jan 2009, at 13:23, Antony Blakey wrote:
> _bulk_docs is the only transactional mechanism in CouchDB. Alarm
> bells went off when I saw that it was effectively decided
I hope I made clear in the past mails that there is no "effective
decision".
Cheers
Jan
--
Re: Current CouchDB state?
Posted by Antony Blakey <an...@gmail.com>.
On 28/01/2009, at 10:35 PM, Jan Lehnardt wrote:
> It sure is, as long as the one person can present his ideas to the
> community which can accept them verbatim. Alas, this has not
> happened yet, so there is nothing to argue about.
Absolutely agree. Right now however the project is moving ahead in
manner that presumes that this *has* happened. That's the basis of my
point. There is no discussion about that vision. Maybe it would get
100% endorsement. Maybe other people might have some ideas that would
adjust or enhance that direction. But right now, there's no chance of
that happening because, as my first email on this point said: Where's
the roadmap, the architectural discussion?
The vision hasn't been handed over to the community. I don't think
it's unreasonable for me to expect that it will be now that it's an
Apache project, and certainly before e.g. an API freeze.
> I acknowledge though that I might have closed the issue a little
> prematurely and without enough context in my spree to find
> 0.9 blockers.
_bulk_docs is the only transactional mechanism in CouchDB. Alarm bells
went off when I saw that it was effectively decided to remove that
behaviour. And even in a cluster setup (as opposed to a partition-
tolerant distributed system), it is possible to provide transactional
behaviour. Sure, it's some work, with it's own set of issue, but IMO
the benefit is enormous, and virtually impossible to do outside the
system unless it provides something like an XA interface, which seems
wrong.
Is there some community discussion about whether CouchDB multinode
clusters are going to be partition tolerant?
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
The greatest challenge to any thinker is stating the problem in a way
that will allow a solution
-- Bertrand Russell
Re: Current CouchDB state?
Posted by Jan Lehnardt <ja...@apache.org>.
>>
> And now it's an Apache project, so it's no longer appropriate for it
> to be one person's vision.
It sure is, as long as the one person can present his ideas to the
community which can accept them verbatim. Alas, this has not
happened yet, so there is nothing to argue about.
I acknowledge though that I might have closed the issue a little
prematurely and without enough context in my spree to find
0.9 blockers.
Cheers
Jan
--
Re: Current CouchDB state?
Posted by Antony Blakey <an...@gmail.com>.
On 28/01/2009, at 10:18 PM, Kerr Rainey wrote:
> 2009/1/28 Antony Blakey <an...@gmail.com>:
>>>> And furthermore, where's the community discussion about this?
>>>
>>> I think this might be the community discussion.
>>
>> Rather after the fact, isn't it?
>
> CouchDB has assumed multiple nodes from it's inception. So any
> discussion that alters that basic principal is going to be after the
> fact.
The fact in this case is the resolution of this issue: https://issues.apache.org/jira/browse/COUCHDB-188
See the dev m/l for some context.
> The project wasn't set up as blank page with a group of people
> discussing what they wanted to do, it was a single persons vision. A
> vision that other people thought was cool and have come together to
> make it better.
And now it's an Apache project, so it's no longer appropriate for it
to be one person's vision.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Only two things are infinite, the universe and human stupidity, and
I'm not sure about the former.
-- Albert Einstein
Re: Current CouchDB state?
Posted by Kerr Rainey <ke...@gmail.com>.
2009/1/28 Antony Blakey <an...@gmail.com>:
>>> And furthermore, where's the community discussion about this?
>>
>> I think this might be the community discussion.
>
> Rather after the fact, isn't it?
CouchDB has assumed multiple nodes from it's inception. So any
discussion that alters that basic principal is going to be after the
fact. The project wasn't set up as blank page with a group of people
discussing what they wanted to do, it was a single persons vision. A
vision that other people thought was cool and have come together to
make it better.
Jumping all over principal assumptions of the platform and then
complaining that there has been no discussion is just silly. It's not
that this can't be discussed, I'm sure everyone here is open to any
discussion on any aspect of CouchDB, but it does have momentum you may
have to overcome.
So we can have the discussion, but don't get bent out of shape just
because the current strategy hasn't been discussed when that strategy
was brought into the project from the start.
--
Kerr
Re: Current CouchDB state?
Posted by Jan Lehnardt <ja...@apache.org>.
On 28 Jan 2009, at 12:33, Antony Blakey wrote:
>
> On 28/01/2009, at 9:58 PM, Chris Anderson wrote:
>
>> On Wed, Jan 28, 2009 at 3:21 AM, Antony Blakey <antony.blakey@gmail.com
>> > wrote:
>>>
>>> And furthermore, where's the community discussion about this?
>>
>> I think this might be the community discussion.
>
> Rather after the fact, isn't it?
See my last mail, but there's no code on this yet and planned for
later. All this comes
back to Damien discussing possible future work on IRC and saying that
single-node
transactions *might* go in favour of a consistent interface (cf. Noah).
There is no definite decision.
Cheers
Jan
--
Re: Current CouchDB state?
Posted by Antony Blakey <an...@gmail.com>.
On 28/01/2009, at 9:58 PM, Chris Anderson wrote:
> On Wed, Jan 28, 2009 at 3:21 AM, Antony Blakey <antony.blakey@gmail.com
> > wrote:
>>
>> And furthermore, where's the community discussion about this?
>
> I think this might be the community discussion.
Rather after the fact, isn't it?
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
The trouble with the world is that the stupid are cocksure and the
intelligent are full of doubt.
-- Bertrand Russell
Re: Current CouchDB state?
Posted by Chris Anderson <jc...@apache.org>.
On Wed, Jan 28, 2009 at 3:21 AM, Antony Blakey <an...@gmail.com> wrote:
>
> On 28/01/2009, at 9:44 PM, Antony Blakey wrote:
>
>>
>> On 28/01/2009, at 9:33 PM, Noah Slater wrote:
>>
>>> On Wed, Jan 28, 2009 at 09:30:19PM +1030, Antony Blakey wrote:
>>>>
>>>> On 28/01/2009, at 9:10 PM, Jan Lehnardt wrote:
>>>>
>>>>>
>>>>> On 28 Jan 2009, at 11:31, Brian Candler wrote:
>>>>>
>>>>>> BTW, I do think the atomic nature of bulk_docs is useful and should
>>>>>> be kept,
>>>>>> as it's the only way to get "transaction" semantics at the moment.
>>>>>
>>>>> We won't be able to guarantee transactions in a multi-node setup.
>>>>
>>>> And there's a universe of single-node applications.
>>>
>>> I would prefer a predictable interface over single-node special-casing.
>>
>> And I would like a transactional guarantee. Why not provide transactional
>> APIs that throw an exception in a multi-node setup? A single node is a
>> useful and IMO common use-case. Possible more common that a multi-node
>> setup.
>>
>> Why penalize such a setup when both can be accommodated?
>
> And furthermore, where's the community discussion about this?
I think this might be the community discussion.
--
Chris Anderson
http://jchris.mfdz.com
Re: Current CouchDB state?
Posted by Noah Slater <ns...@apache.org>.
On Wed, Jan 28, 2009 at 10:24:33PM +1030, Antony Blakey wrote:
>
> On 28/01/2009, at 10:10 PM, Jan Lehnardt wrote:
>
>>
>> On 28 Jan 2009, at 12:21, Antony Blakey wrote:
>>
>>> And furthermore, where's the community discussion about this?
>>> Where's the roadmap, the architectural discussion? Or are we all
>>> meant to stand back in awe of one person's god-like vision,
>>> unexplained, undiscussed, ineffable? That doesn't sound like an
>>> Apache project.
>>
>> Easy now. CouchDB and the community is balancing making a make-your-
>> own-cloud high performance multi-node database and a single-node-
>> standalone-application database. There are forces that pull in one and
>> others that pull in the, well, other direction. I think this is good.
>
> My point isn't about this or any other specific technical issue.
>
> This project seems to be administered in accordance with Damien's
> unexplained and undiscussed plan. That would be fine, except that this
> is an Apache project. I'm looking for a discussion of the plan, of the
> goals, of the future architecture, rather than merely being allowed to
> discuss some technical details as CouchDB is built according to some
> private vision.
I think you're over-reacting, again.
I find the way that you've framed this objectionable, and it offends me that
you've made such wild accusations about the mismanagement of the PMC. It would
really smooth the consensus building and community process if you were able to
give us the benefit of the doubt.
This whole sub-thread was a reaction to my off-the-cuff remark. Absurd.
--
Noah Slater, http://tumbolia.org/nslater
Re: Current CouchDB state?
Posted by Jan Lehnardt <ja...@apache.org>.
On 28 Jan 2009, at 12:54, Antony Blakey wrote:
>
> On 28/01/2009, at 10:10 PM, Jan Lehnardt wrote:
>
>>
>> On 28 Jan 2009, at 12:21, Antony Blakey wrote:
>>
>>> And furthermore, where's the community discussion about this?
>>> Where's the roadmap, the architectural discussion? Or are we all
>>> meant to stand back in awe of one person's god-like vision,
>>> unexplained, undiscussed, ineffable? That doesn't sound like an
>>> Apache project.
>>
>> Easy now. CouchDB and the community is balancing making a make-your-
>> own-cloud high performance multi-node database and a single-node-
>> standalone-application database. There are forces that pull in one
>> and others that pull in the, well, other direction. I think this is
>> good.
>
> My point isn't about this or any other specific technical issue.
>
> This project seems to be administered in accordance with Damien's
> unexplained and undiscussed plan. That would be fine, except that
> this is an Apache project. I'm looking for a discussion of the plan,
> of the goals, of the future architecture, rather than merely being
> allowed to discuss some technical details as CouchDB is built
> according to some private vision.
He made a comment on IRC (wich we all acknowledge is *NOT* an official
channel for the transparent development of CouchDB (only for day-to-
day issues for us and users). He didn't make any decision. He said we
*might* change something. There has been no discussion on any ML, by
ASF statues, this means there is no decision. Is it me, or are you
being dense? (No offence).
It is good that we get controlled by the community, and appreciate
your input Antony, but I think you are taking this a bit out of
proportion.
See Chris changing the name of _temp_views because he though we agreed
on that months ago. No discussion and actually changes int trunk. Now:
A comment on IRC.
Cheers
Jan
--
Re: Current CouchDB state?
Posted by Antony Blakey <an...@gmail.com>.
On 28/01/2009, at 10:10 PM, Jan Lehnardt wrote:
>
> On 28 Jan 2009, at 12:21, Antony Blakey wrote:
>
>> And furthermore, where's the community discussion about this?
>> Where's the roadmap, the architectural discussion? Or are we all
>> meant to stand back in awe of one person's god-like vision,
>> unexplained, undiscussed, ineffable? That doesn't sound like an
>> Apache project.
>
> Easy now. CouchDB and the community is balancing making a make-your-
> own-cloud high performance multi-node database and a single-node-
> standalone-application database. There are forces that pull in one
> and others that pull in the, well, other direction. I think this is
> good.
My point isn't about this or any other specific technical issue.
This project seems to be administered in accordance with Damien's
unexplained and undiscussed plan. That would be fine, except that this
is an Apache project. I'm looking for a discussion of the plan, of the
goals, of the future architecture, rather than merely being allowed to
discuss some technical details as CouchDB is built according to some
private vision.
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
There are two ways of constructing a software design: One way is to
make it so simple that there are obviously no deficiencies, and the
other way is to make it so complicated that there are no obvious
deficiencies.
-- C. A. R. Hoare
Re: Current CouchDB state?
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 28, 2009, at 6:40 AM, Jan Lehnardt wrote:
>
> On 28 Jan 2009, at 12:21, Antony Blakey wrote:
>
>> And furthermore, where's the community discussion about this?
>> Where's the roadmap, the architectural discussion? Or are we all
>> meant to stand back in awe of one person's god-like vision,
>> unexplained, undiscussed, ineffable? That doesn't sound like an
>> Apache project.
>
> Easy now. CouchDB and the community is balancing making a make-your-
> own-cloud high performance multi-node database and a single-node-
> standalone-application database. There are forces that pull in one
> and others that pull in the, well, other direction. I think this is
> good.
This *is* good, an natural.
>
>
> The partitioning feature, to all of this alludes, is not planned for
> 1.0. There is plenty of time to discuss this. I see the need for
> both single-node transactions and single-node fire-and-forget bulk
> inserts.
I think the issue is that Antony, who I've seen is an active and
attentive member of the communty, appears surprised by this
information - not with the detail, but rather that such decisions were
made, and come as a surprise.
>
> CouchDB is also about giving people the right ideas to build
> scalable systems. We could implement auto-increment on a single node
> and throw exceptions when you try to use it in the multi-node
> environment. But we don't. We teach people why auto-increment is a
> bad idea in the first place. Bulk inserts are a different beast in
> that they are genuinely useful in a single-node setup. But if you
> use single-node transactions, we cannot give you the "scaling
> included" guarantee that you get when you don't rely on single-node
> transactions. But that is only one force that pulls. See above.
>
> I believe we should take this to dev@ when the database partitioning
> feature is being discussed (after 0.9, I hope). The only thing to
> discuss now is if we want fire-and-forget bulk inserts in 0.9.
> Although, come to think of it, we don't want to remove single-node
> transactions after 0.9.
>
> So: Fire away on dev@.
Is there anything in the dev@ archive on this to catch up on? I'm
interested in the same subject. It would be nice if the
implementation detail of the # of servers in the cluster didn't change
basic functionality.
geir
Re: Current CouchDB state?
Posted by Jan Lehnardt <ja...@apache.org>.
On 28 Jan 2009, at 12:21, Antony Blakey wrote:
> And furthermore, where's the community discussion about this?
> Where's the roadmap, the architectural discussion? Or are we all
> meant to stand back in awe of one person's god-like vision,
> unexplained, undiscussed, ineffable? That doesn't sound like an
> Apache project.
Easy now. CouchDB and the community is balancing making a make-your-
own-cloud high performance multi-node database and a single-node-
standalone-application database. There are forces that pull in one and
others that pull in the, well, other direction. I think this is good.
The partitioning feature, to all of this alludes, is not planned for
1.0. There is plenty of time to discuss this. I see the need for both
single-node transactions and single-node fire-and-forget bulk inserts.
CouchDB is also about giving people the right ideas to build scalable
systems. We could implement auto-increment on a single node and throw
exceptions when you try to use it in the multi-node environment. But
we don't. We teach people why auto-increment is a bad idea in the
first place. Bulk inserts are a different beast in that they are
genuinely useful in a single-node setup. But if you use single-node
transactions, we cannot give you the "scaling included" guarantee that
you get when you don't rely on single-node transactions. But that is
only one force that pulls. See above.
I believe we should take this to dev@ when the database partitioning
feature is being discussed (after 0.9, I hope). The only thing to
discuss now is if we want fire-and-forget bulk inserts in 0.9.
Although, come to think of it, we don't want to remove single-node
transactions after 0.9.
So: Fire away on dev@.
Cheers
Jan
--
Re: Current CouchDB state?
Posted by Antony Blakey <an...@gmail.com>.
On 28/01/2009, at 9:44 PM, Antony Blakey wrote:
>
> On 28/01/2009, at 9:33 PM, Noah Slater wrote:
>
>> On Wed, Jan 28, 2009 at 09:30:19PM +1030, Antony Blakey wrote:
>>>
>>> On 28/01/2009, at 9:10 PM, Jan Lehnardt wrote:
>>>
>>>>
>>>> On 28 Jan 2009, at 11:31, Brian Candler wrote:
>>>>
>>>>> BTW, I do think the atomic nature of bulk_docs is useful and
>>>>> should
>>>>> be kept,
>>>>> as it's the only way to get "transaction" semantics at the moment.
>>>>
>>>> We won't be able to guarantee transactions in a multi-node setup.
>>>
>>> And there's a universe of single-node applications.
>>
>> I would prefer a predictable interface over single-node special-
>> casing.
>
> And I would like a transactional guarantee. Why not provide
> transactional APIs that throw an exception in a multi-node setup? A
> single node is a useful and IMO common use-case. Possible more
> common that a multi-node setup.
>
> Why penalize such a setup when both can be accommodated?
And furthermore, where's the community discussion about this? Where's
the roadmap, the architectural discussion? Or are we all meant to
stand back in awe of one person's god-like vision, unexplained,
undiscussed, ineffable? That doesn't sound like an Apache project.
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Every task involves constraint,
Solve the thing without complaint;
There are magic links and chains
Forged to loose our rigid brains.
Structures, structures, though they bind,
Strangely liberate the mind.
-- James Fallen
Re: Current CouchDB state?
Posted by Antony Blakey <an...@gmail.com>.
On 28/01/2009, at 9:33 PM, Noah Slater wrote:
> On Wed, Jan 28, 2009 at 09:30:19PM +1030, Antony Blakey wrote:
>>
>> On 28/01/2009, at 9:10 PM, Jan Lehnardt wrote:
>>
>>>
>>> On 28 Jan 2009, at 11:31, Brian Candler wrote:
>>>
>>>> BTW, I do think the atomic nature of bulk_docs is useful and should
>>>> be kept,
>>>> as it's the only way to get "transaction" semantics at the moment.
>>>
>>> We won't be able to guarantee transactions in a multi-node setup.
>>
>> And there's a universe of single-node applications.
>
> I would prefer a predictable interface over single-node special-
> casing.
And I would like a transactional guarantee. Why not provide
transactional APIs that throw an exception in a multi-node setup? A
single node is a useful and IMO common use-case. Possible more common
that a multi-node setup.
Why penalize such a setup when both can be accommodated?
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
A Buddhist walks up to a hot-dog stand and says, "Make me one with
everything". He then pays the vendor and asks for change. The vendor
says, "Change comes from within".
Re: Current CouchDB state?
Posted by Noah Slater <ns...@apache.org>.
On Wed, Jan 28, 2009 at 09:30:19PM +1030, Antony Blakey wrote:
>
> On 28/01/2009, at 9:10 PM, Jan Lehnardt wrote:
>
>>
>> On 28 Jan 2009, at 11:31, Brian Candler wrote:
>>
>>> BTW, I do think the atomic nature of bulk_docs is useful and should
>>> be kept,
>>> as it's the only way to get "transaction" semantics at the moment.
>>
>> We won't be able to guarantee transactions in a multi-node setup.
>
> And there's a universe of single-node applications.
I would prefer a predictable interface over single-node special-casing.
--
Noah Slater, http://tumbolia.org/nslater
Re: Current CouchDB state?
Posted by Antony Blakey <an...@gmail.com>.
On 28/01/2009, at 9:10 PM, Jan Lehnardt wrote:
>
> On 28 Jan 2009, at 11:31, Brian Candler wrote:
>
>> BTW, I do think the atomic nature of bulk_docs is useful and should
>> be kept,
>> as it's the only way to get "transaction" semantics at the moment.
>
> We won't be able to guarantee transactions in a multi-node setup.
And there's a universe of single-node applications.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
The trouble with the world is that the stupid are cocksure and the
intelligent are full of doubt.
-- Bertrand Russell
Re: Current CouchDB state?
Posted by Jan Lehnardt <ja...@apache.org>.
On 28 Jan 2009, at 11:31, Brian Candler wrote:
> BTW, I do think the atomic nature of bulk_docs is useful and should
> be kept,
> as it's the only way to get "transaction" semantics at the moment.
We won't be able to guarantee transactions in a multi-node setup.
Cheers
Jan
--
Re: Current CouchDB state?
Posted by Brian Candler <B....@pobox.com>.
On Wed, Jan 28, 2009 at 01:32:30AM -0800, Chris Anderson wrote:
> There is an option to delay and batch writes. There has been recent
> changes in the source around this area. I think the default puts
> performance ahead of absolute consistency, but for critical apps you
> should use X-Couch-Full-Commit: true
OK, I'll have a look round so more to try to understand what's going on.
What would be ideal for me is a commit-with-wait option: e.g. wait up to
100ms before committing and responding. This would allow other requests
arriving within that window to be combined and all of them safely committed
together.
Any particular reason to use a HTTP header rather than a query string
parameter? Most other features seem to be controlled in the latter way.
BTW, I do think the atomic nature of bulk_docs is useful and should be kept,
as it's the only way to get "transaction" semantics at the moment.
Regards,
Brian.
Re: Current CouchDB state?
Posted by Chris Anderson <jc...@apache.org>.
On Wed, Jan 28, 2009 at 1:21 AM, Brian Candler <B....@pobox.com> wrote:
> On Wed, Jan 28, 2009 at 09:05:20AM +0000, Brian Candler wrote:
>> Again looking at source, I see two very different paths in couch_db.erl
>> depending on the presence of that magic header:
>>
>> update_docs(Db, Docs, Options, false) -> ...
>> update_docs(Db, Docs, Options, true) -> ...
>
> Sorry, that was a mistake. It's the Options parameter which contains [] or
> [full_commit]; the call from couch_http_db calls update_docs/3, and the
> fourth argument defaults to true.
>
> I'm not sure what the false version of this function does - something to do
> with not allocating new revs? - but it's not relevant in this case.
>
It's cool that you are digging in like this. The false option is used
by the replication API.
> If there are any uuid conflicts then the whole
> batch will fail, and I'll have to patch it up and resubmit the whole lot.
The transactional nature of bulk_docs may not continue to hold, or
there may be an option to continue inserting even if some docs fail,
and then return the status of each doc as a list.
re: {minor_version,1}
I remember reading about that feature. I'm not sure what the arguments
are on either side. So far we haven't seen any reason to worry about
our number handling in Erlang. Maybe someone out there has a good
argument for switching to it?
> if multiple clients are PUTting individual documents, does a
> write-sync-write-sync take place for each one?
Writes to disk are synchronous (eg one writer per file).
There is an option to delay and batch writes. There has been recent
changes in the source around this area. I think the default puts
performance ahead of absolute consistency, but for critical apps you
should use X-Couch-Full-Commit: true
--
Chris Anderson
http://jchris.mfdz.com
Re: Current CouchDB state?
Posted by Brian Candler <B....@pobox.com>.
On Wed, Jan 28, 2009 at 09:05:20AM +0000, Brian Candler wrote:
> Again looking at source, I see two very different paths in couch_db.erl
> depending on the presence of that magic header:
>
> update_docs(Db, Docs, Options, false) -> ...
> update_docs(Db, Docs, Options, true) -> ...
Sorry, that was a mistake. It's the Options parameter which contains [] or
[full_commit]; the call from couch_http_db calls update_docs/3, and the
fourth argument defaults to true.
I'm not sure what the false version of this function does - something to do
with not allocating new revs? - but it's not relevant in this case.
B.
Re: Current CouchDB state?
Posted by Brian Candler <B....@pobox.com>.
On Tue, Jan 27, 2009 at 01:30:37PM -0800, Chris Anderson wrote:
> > - To get such high performance, is it necessary to use _bulk_docs, or was
> > it achieved with regular PUT operations?
>
> This was done with a pure Erlang interface, and bulk size of 1k docs.
> The HTTP interface may add minimal overhead if your json is not
> complex and you use bulk_docs
Aha, using a not-yet-published API is cheating :-) But I expect HTTP
bulk_docs will come close.
The trouble here with bulk_docs is that it's an all-or-nothing interface.
For example: suppose I collect RADIUS packets for 100ms before writing them.
I also use the recommended way of creating new docs by using the _uuids call
first, then submit the batch. If there are any uuid conflicts then the whole
batch will fail, and I'll have to patch it up and resubmit the whole lot.
It's not a big problem in this case, but it means it's not practical to
write a generic "HTTP batching proxy" which sits in front of couchdb and
batches arbitrary requests from a mixture of clients.
(Actually, authorization would be a problem anyway, once validate_doc_update
is widely used)
> > - Does Couchdb commit its data to stable storage *before* returning a HTTP
> > response? That is, once you receive a HTTP success response, you can be
> > sure that the data has already hit the disk?
>
> There is a header you can send which forces a full fsync before it
> returns. In the default case, it only returns after writing the file
Looking through the source, I guess you are talking about
X-Couch-Full-Commit: true
However, I don't understand this in the light of what's written at
http://couchdb.apache.org/docs/overview.html
"When CouchDB documents are updated, all data and associated indexes are
flushed to disk and the transactional commit always leaves the database in a
completely consistent state. Commits occur in two steps:
1. All document data and associated index updates are synchronously
flushed to disk.
2. The updated database header is written in two consecutive, identical
chunks to make up the first 4k of the file, and then synchronously
flushed to disk."
So it seems we have write - sync - write - sync, and a quick look through
the source shows this in couch_file:write_header/3
What I don't yet understand is:
- at what point is the HTTP response sent to the server?
- at what point is this commit cycle done?
- if multiple clients are PUTting individual documents, does a
write-sync-write-sync take place for each one?
Again looking at source, I see two very different paths in couch_db.erl
depending on the presence of that magic header:
update_docs(Db, Docs, Options, false) -> ...
update_docs(Db, Docs, Options, true) -> ...
Both send an update_docs message to the UpdatePid process, but I then got a
bit lost in update_docs_int.
> (but trusts the OS -- which usually lies in the interest of speed.)
Do you mean it carries on after write() returns? The OS doesn't lie, it just
says truthfully that it has stuck your data into dirty buffers in the VFS
:-)
> Couch uses Erlang's term_to_binary for saving, which I believe uses
> gzip by default. This is worth verifying, it's been a while since I
> toured that part of the source.
I found it: couch_file:append_term/2 calls
term_to_binary(Term, [compressed])
The term_to_binary/2 docs say that {compressed,6} is the current default,
i.e. gzip -6
It also mentions that you might want to add {minor_version,1} to allow
floats to be stored as 64-bit doubles rather than in text. This gives both
more accuracy and smaller storage. Otherwise, if json_decode decodes JSON
floats into IEEE, you'll end up converting back to text when it hits the
disk.
Cheers,
Brian.
Re: Current CouchDB state?
Posted by Chris Anderson <jc...@apache.org>.
On Tue, Jan 27, 2009 at 1:22 PM, Brian Candler <B....@pobox.com> wrote:
>> In a recent benchmark: Inserting 100b docs into an empty database:
>> ~6200docs/s. Inserting the same docs in a 90 000 000 doc db: 6000docs/
>> s (with sequential ids). Data scales.
>
> This is very interesting - one of the applications I'm thinking of has a
> profile just like this (warehouse for RADIUS accounting records)
>
> I have a couple of questions relating to this.
>
> - To get such high performance, is it necessary to use _bulk_docs, or was
> it achieved with regular PUT operations?
This was done with a pure Erlang interface, and bulk size of 1k docs.
The HTTP interface may add minimal overhead if your json is not
complex and you use bulk_docs
>
> - Does Couchdb commit its data to stable storage *before* returning a HTTP
> response? That is, once you receive a HTTP success response, you can be
> sure that the data has already hit the disk?
There is a header you can send which forces a full fsync before it
returns. In the default case, it only returns after writing the file
(but trusts the OS -- which usually lies in the interest of speed.)
>
> If Couchdb can handle 6,000 individual PUT requests per second, *and* only
> respond after they are committed to stable storage, then I think it must be
> batching the writes and hence delaying the responses somewhat (by how much?
> I couldn't see a tunable parameter for this)
This involved 6 saves per second, also allowing the OS to manage the disk.
>
> However if this performance is only achievable using _bulk_docs, I'll have
> to write my RADIUS server / Couchdb client to perform its own local
> batching and POST these batches a few times per second.
>
> I presume that batching also affects disk space used (before compaction) - I
> wouldn't want each 200 byte RADIUS record taking up 4KB :-)
>
> Final question: does Couchdb perform any gzip-like compression when writing
> the JSON to disk? These 200 byte RADIUS records will become a lot larger
> when expanded into verbose JSON.
Couch uses Erlang's term_to_binary for saving, which I believe uses
gzip by default. This is worth verifying, it's been a while since I
toured that part of the source.
>
> {
> "framed_ip_address":"192.168.1.1", // 6 bytes in original packet
> ... etc
> }
>
> Regards,
>
> Brian.
>
--
Chris Anderson
http://jchris.mfdz.com
Re: [user] Re: Current CouchDB state?
Posted by Wout Mertens <wm...@cisco.com>.
On Jan 27, 2009, at 8:43 PM, Jan Lehnardt wrote:
>> Finally, I know that the replication works. Is there a "standard"
>> way of using it with supporting programs?
>
> What do you mean? CouchDB is the only program (sadly) that supports
> CouchDB replication.
Err to steer the conversation back to the original subject, I meant,
are there scripts that do the replication for you in common scenarios?
Or is everything still roll-your-own?
I can see these problems in replication:
- Keeping track of replicas
- Handling replica-down and replica-up events
- Keeping track of which subset of data to replicate
I suppose a blind "try until it succeeds" approach might work, so a
"while sleep 3; do curl URL_TO_DO_REPLICATION; done" might be the end
of it.
I'm just wondering.
Wout.
Re: Current CouchDB state?
Posted by Brian Candler <B....@pobox.com>.
> In a recent benchmark: Inserting 100b docs into an empty database:
> ~6200docs/s. Inserting the same docs in a 90 000 000 doc db: 6000docs/
> s (with sequential ids). Data scales.
This is very interesting - one of the applications I'm thinking of has a
profile just like this (warehouse for RADIUS accounting records)
I have a couple of questions relating to this.
- To get such high performance, is it necessary to use _bulk_docs, or was
it achieved with regular PUT operations?
- Does Couchdb commit its data to stable storage *before* returning a HTTP
response? That is, once you receive a HTTP success response, you can be
sure that the data has already hit the disk?
If Couchdb can handle 6,000 individual PUT requests per second, *and* only
respond after they are committed to stable storage, then I think it must be
batching the writes and hence delaying the responses somewhat (by how much?
I couldn't see a tunable parameter for this)
However if this performance is only achievable using _bulk_docs, I'll have
to write my RADIUS server / Couchdb client to perform its own local
batching and POST these batches a few times per second.
I presume that batching also affects disk space used (before compaction) - I
wouldn't want each 200 byte RADIUS record taking up 4KB :-)
Final question: does Couchdb perform any gzip-like compression when writing
the JSON to disk? These 200 byte RADIUS records will become a lot larger
when expanded into verbose JSON.
{
"framed_ip_address":"192.168.1.1", // 6 bytes in original packet
... etc
}
Regards,
Brian.
Re: Current CouchDB state?
Posted by Jan Lehnardt <ja...@apache.org>.
On 27 Jan 2009, at 14:38, Wout Mertens wrote:
> Hi there,
>
> I'm a big fan of CouchDB and I want to use it in production but I'm
> holding out for a somewhat stable release. In preparation however I
> would like to know where CouchDB stands in High Availability and/or
> Scalability.
In a recent benchmark: Inserting 100b docs into an empty database:
~6200docs/s. Inserting the same docs in a 90 000 000 doc db: 6000docs/
s (with sequential ids). Data scales.
Scaling over multiple nodes needs to be done on the client side for
now. We're looking into database partitioning.
HA: It's HTTP. If you know how to make HTTP highly available, there's
your receipt.
> I know that Erlang promises that you can run a single program across
> multiple CPUs and even systems at the same time, and you can upgrade
> the program while it is handling requests. I presume Erlang also
> handles the case where a system goes offline?
Erlang gives you the tools to build these systems. You still have to
deal with outages yourself (that is us), because that is application
specific. We don't distribute over multiple nodes per instance yet.
> Furthermore, I know that the view server runs separately from the
> CouchDB process, and it can in principle split the map and reduce
> phases across many compute nodes.
Yes, it runs separately and yes, with a little effort you could
distribute the computation, but the sweet spot for M/R is heavy
computation and little data movement where views are lightweight
computing on loads of data. So there's little to gain if your data
isn't already on the remote systems. Database partitioning will change
that game.
> Finally, I know that the replication works. Is there a "standard"
> way of using it with supporting programs?
What do you mean? CouchDB is the only program (sadly) that supports
CouchDB replication.
> So my question is: How much of this is already a reality? Is any
> testing being done for these scenarios? Is this on the roadmap for
> 1.0?
People are using CouchDB in production with success now (and have been
for over a year). Depending on your app, CouchDB might be "there" for
you already.
Feel free to send in more specific questions. :)
Cheers
Jan
--
Re: Current CouchDB state?
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Jan 27, 2009 at 8:38 AM, Wout Mertens <wm...@cisco.com> wrote:
> Hi there,
>
> I'm a big fan of CouchDB and I want to use it in production but I'm holding
> out for a somewhat stable release. In preparation however I would like to
> know where CouchDB stands in High Availability and/or Scalability.
>
> I know that Erlang promises that you can run a single program across
> multiple CPUs and even systems at the same time, and you can upgrade the
> program while it is handling requests. I presume Erlang also handles the
> case where a system goes offline?
>
> Furthermore, I know that the view server runs separately from the CouchDB
> process, and it can in principle split the map and reduce phases across many
> compute nodes.
>
> Finally, I know that the replication works. Is there a "standard" way of
> using it with supporting programs?
>
>
> So my question is: How much of this is already a reality? Is any testing
> being done for these scenarios? Is this on the roadmap for 1.0?
>
> Thanks,
>
> Wout.
>
>
Just thought I'd mention that this article came across my google
alerts today and it's actually related to the original thread:
http://www.joeandmotorboat.com/2009/01/27/couchdb-load-balancing-and-replication-using-haproxy/
There are people doing some fault tolerant multi-node setups. The
different articles I've read all tend to have one guiding theme, "Holy
shit, that was easy!"
Anyway, I think the best way to figure out if CouchDB is up to snuff
for what you're wanting, the best thing is to setup some simulated
load and see if shit hits the fan. :) We're always excited to hear
numbers on different load types and configurations.
HTH,
Paul Davis