You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Adam Kocoloski <ad...@gmail.com> on 2009/03/05 02:34:42 UTC
I got a little carried away ...
I was working through some replication tickets this week and thinking
more and more that the replicator could benefit from being
restructured along OTP principles. So, I went ahead and did it.
Here's the structure I've worked out so far:
http://github.com/kocolosk/couchdb/tree/otpify-replication
* couch_primary_services gets a new child
(couch_replication_supervisor) which is itself a supervisor using a
one_for_one strategy.
* Replications are modeled as gen_servers and added as transient
children of couch_rep_sup. couch_rep_sup ensures that identical
replications do not run simultaneously.
* The gen_server spawn_links a separate process that enumerates the
docs on the source and makes synchronous calls to the gen_server
telling it to copy the documents that don't already exist on the target.
* Opening and saving of documents and attachments is handled
internally by the gen_server. I'm hoping that concentrating all this
activity in the server will help us do a better job of pipeline and
memory management.
Sorry for not discussing a change this big ahead of time. Damien, I
saw that you had some recent checkins related to replication security;
I'd be happy to merge those in. The replication API itself has not
changed at all.
This branch is still work-in-progress, but in my opinion it's almost
as fast as and more robust than the code in trunk right now. If it
meets with peoples' approval and we can get some feedback from the
various reporters in JIRA, it might make sense to tag it for 0.9.
Cheers,
Adam
Re: I got a little carried away ...
Posted by Jan Lehnardt <ja...@apache.org>.
Thanks Adam, this looks promising!
On 5 Mar 2009, at 02:34, Adam Kocoloski wrote:
> I was working through some replication tickets this week and
> thinking more and more that the replicator could benefit from being
> restructured along OTP principles. So, I went ahead and did it.
> Here's the structure I've worked out so far:
>
> http://github.com/kocolosk/couchdb/tree/otpify-replication
>
> * couch_primary_services gets a new child
> (couch_replication_supervisor) which is itself a supervisor using a
> one_for_one strategy.
>
> * Replications are modeled as gen_servers and added as transient
> children of couch_rep_sup. couch_rep_sup ensures that identical
> replications do not run simultaneously.
>
> * The gen_server spawn_links a separate process that enumerates the
> docs on the source and makes synchronous calls to the gen_server
> telling it to copy the documents that don't already exist on the
> target.
>
> * Opening and saving of documents and attachments is handled
> internally by the gen_server. I'm hoping that concentrating all
> this activity in the server will help us do a better job of pipeline
> and memory management.
>
> Sorry for not discussing a change this big ahead of time. Damien, I
> saw that you had some recent checkins related to replication
> security; I'd be happy to merge those in. The replication API
> itself has not changed at all.
The worst that can happen is that we don't like that patch and you
worked for nothing. As long as you (or anyone) don't (doesn't) simply
commit a major patch to trunk without discussion, writing patches
without discussion is just fine :)
> This branch is still work-in-progress, but in my opinion it's almost
> as fast as and more robust than the code in trunk right now. If it
> meets with peoples' approval and we can get some feedback from the
> various reporters in JIRA, it might make sense to tag it for 0.9.
> Cheers,
What are the open issues with that branch? How can we help?
Cheers
Jan
--
Re: I got a little carried away ...
Posted by Damien Katz <da...@apache.org>.
On Mar 4, 2009, at 8:34 PM, Adam Kocoloski wrote:
> I was working through some replication tickets this week and
> thinking more and more that the replicator could benefit from being
> restructured along OTP principles. So, I went ahead and did it.
> Here's the structure I've worked out so far:
>
> http://github.com/kocolosk/couchdb/tree/otpify-replication
>
> * couch_primary_services gets a new child
> (couch_replication_supervisor) which is itself a supervisor using a
> one_for_one strategy.
>
> * Replications are modeled as gen_servers and added as transient
> children of couch_rep_sup. couch_rep_sup ensures that identical
> replications do not run simultaneously.
>
> * The gen_server spawn_links a separate process that enumerates the
> docs on the source and makes synchronous calls to the gen_server
> telling it to copy the documents that don't already exist on the
> target.
>
> * Opening and saving of documents and attachments is handled
> internally by the gen_server. I'm hoping that concentrating all
> this activity in the server will help us do a better job of pipeline
> and memory management.
>
This sounds good.
> Sorry for not discussing a change this big ahead of time.
We are discussing it now :)
> Damien, I saw that you had some recent checkins related to
> replication security; I'd be happy to merge those in. The
> replication API itself has not changed at all.
Let me know if you need any help merging.
>
> This branch is still work-in-progress, but in my opinion it's almost
> as fast as and more robust than the code in trunk right now. If it
> meets with peoples' approval and we can get some feedback from the
> various reporters in JIRA, it might make sense to tag it for 0.9.
> Cheers,
This work will also be useful for setting up continuos replications.
Soon we'll have Comet events to notify a waiting replicator of changes
to replicate, so change replicate as soon as they happen, not just on
a schedule.
-Damien
Re: I got a little carried away ...
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Mar 6, 2009 at 12:08 PM, Noah Slater <ns...@apache.org> wrote:
> On Fri, Mar 06, 2009 at 09:55:19AM -0600, Zachary Zolton wrote:
>> >> Actually, the most time-intensive part, for me, is installing Erlang
>> >> from source. Perhaps I should be bugging some Ubuntu folks for that?!
>> >
>> > Eh!?
>>
>> I should probably create my own binary package to speed up my server
>> deployments. As I'm on EC2, I've found it most convenient to create
>> new servers from scratch, but I don't think this is the best place to
>> discuss.
>
> Was wondering why you would compile Erlang, when you can get a package.
>
At one point the Ubuntu package was the notorious Erlang 5.5.5 VM
that's got a bug that caused CouchDB to run fairly slow. Just in case
anyone has still got that version lying around.
HTH,
Paul Davis
> --
> Noah Slater, http://tumbolia.org/nslater
>
Re: I got a little carried away ...
Posted by Noah Slater <ns...@apache.org>.
On Fri, Mar 06, 2009 at 09:55:19AM -0600, Zachary Zolton wrote:
> >> Actually, the most time-intensive part, for me, is installing Erlang
> >> from source. Perhaps I should be bugging some Ubuntu folks for that?!
> >
> > Eh!?
>
> I should probably create my own binary package to speed up my server
> deployments. As I'm on EC2, I've found it most convenient to create
> new servers from scratch, but I don't think this is the best place to
> discuss.
Was wondering why you would compile Erlang, when you can get a package.
--
Noah Slater, http://tumbolia.org/nslater
Re: I got a little carried away ...
Posted by Zachary Zolton <za...@gmail.com>.
@noah
> Huh!? You've been able to apt-get install couchdb for over a year now!
Not 0.9. (^_-)
>> Actually, the most time-intensive part, for me, is installing Erlang
>> from source. Perhaps I should be bugging some Ubuntu folks for that?!
>
> Eh!?
I should probably create my own binary package to speed up my server
deployments. As I'm on EC2, I've found it most convenient to create
new servers from scratch, but I don't think this is the best place to
discuss.
So, sorry again to have opened my big, newb-ish mouth!
—Zach
Re: I got a little carried away ...
Posted by Noah Slater <ns...@apache.org>.
On Thu, Mar 05, 2009 at 04:32:29PM -0600, Zachary Zolton wrote:
> Just waiting anxiously for the time when I can simply apt-get install couchdb!!
Huh!? You've been able to apt-get install couchdb for over a year now!
> Actually, the most time-intensive part, for me, is installing Erlang
> from source. Perhaps I should be bugging some Ubuntu folks for that?!
Eh!?
--
Noah Slater, http://tumbolia.org/nslater
Re: I got a little carried away ...
Posted by Zachary Zolton <za...@gmail.com>.
Just waiting anxiously for the time when I can simply apt-get install couchdb!!
Actually, the most time-intensive part, for me, is installing Erlang
from source. Perhaps I should be bugging some Ubuntu folks for that?!
Cheers,
Zach
On Thu, Mar 5, 2009 at 4:07 PM, Noah Slater <ns...@apache.org> wrote:
> On Thu, Mar 05, 2009 at 01:49:59PM -0600, Zachary Zolton wrote:
>> @Noah, my apologies; I didn't mean to trivialize your labor.
>
> ... and sorry if I came across as grumpy!
>
> --
> Noah Slater, http://tumbolia.org/nslater
>
Re: I got a little carried away ...
Posted by Noah Slater <ns...@apache.org>.
On Thu, Mar 05, 2009 at 01:49:59PM -0600, Zachary Zolton wrote:
> @Noah, my apologies; I didn't mean to trivialize your labor.
... and sorry if I came across as grumpy!
--
Noah Slater, http://tumbolia.org/nslater
Re: I got a little carried away ...
Posted by Zachary Zolton <za...@gmail.com>.
> I am the Debian/Ubuntu package maintainer, and CouchDB's release manager.
>
> Sergei looks after the Erlang OTP, and the current version is 1:12.b.5-dfsg-2.
>
> Making a release is non-trivial for me, I will say again.
>
> If we're going to make a release in a few weeks, we should wait for that.
>
> --
> Noah Slater, http://tumbolia.org/nslater
>
@Noah, my apologies; I didn't mean to trivialize your labor.
—Zach
Re: I got a little carried away ...
Posted by Noah Slater <ns...@apache.org>.
On Thu, Mar 05, 2009 at 11:00:23AM -0600, Zachary Zolton wrote:
> Not to mention starting the process to nudge Debian/Ubuntu package
> maintainers to get a sufficiently-recent version of Erlang out there!
> Apt is telling me that 1:11.b.5dfsg-11 is what's currently
> available...
I am the Debian/Ubuntu package maintainer, and CouchDB's release manager.
Sergei looks after the Erlang OTP, and the current version is 1:12.b.5-dfsg-2.
Making a release is non-trivial for me, I will say again.
If we're going to make a release in a few weeks, we should wait for that.
--
Noah Slater, http://tumbolia.org/nslater
Re: I got a little carried away ...
Posted by Zachary Zolton <za...@gmail.com>.
Not to mention starting the process to nudge Debian/Ubuntu package
maintainers to get a sufficiently-recent version of Erlang out there!
Apt is telling me that 1:11.b.5dfsg-11 is what's currently
available...
On Thu, Mar 5, 2009 at 10:44 AM, Kerr Rainey <ke...@gmail.com> wrote:
> 2009/3/5 Noah Slater <ns...@apache.org>:
>> /me puts on his release manager hat
>>
>> Delay, cut 0.9 with the patches. Making a release is non-trivial.
>
> I thought the point of trying to get a 0.9 release out was to have a
> realse of all the stuff that's been sitting in trunk for a while?
> Shouldn't we just get it out. It doesn't mean that we then need to do
> another release in 2 weeks. There will always be new work that is 2
> weeks away that could push back a release. At some stage something
> isn't going to make it and it will go into the next cut.
>
> But, you know, it's not like I'm the one that has to do any work. I
> just think it would be good to get a release out.
>
> --
> Kerr
>
Re: I got a little carried away ...
Posted by Jan Lehnardt <ja...@apache.org>.
On 5 Mar 2009, at 17:44, Kerr Rainey wrote:
> 2009/3/5 Noah Slater <ns...@apache.org>:
>> /me puts on his release manager hat
>>
>> Delay, cut 0.9 with the patches. Making a release is non-trivial.
>
> I thought the point of trying to get a 0.9 release out was to have a
> realse of all the stuff that's been sitting in trunk for a while?
Yeah, but that was when the patch for 280 was still in the works, the
rep_security branch still in the trenches and Adam's replication fix
not on the horizon.
> Shouldn't we just get it out. It doesn't mean that we then need to do
> another release in 2 weeks. There will always be new work that is 2
> weeks away that could push back a release. At some stage something
> isn't going to make it and it will go into the next cut.
>
> But, you know, it's not like I'm the one that has to do any work. I
> just think it would be good to get a release out.
I wouldn't wann to ask Noah to release twice in two weeks if we can
avoid it.
+1 one on merging all three patches (pending their state considered by
their authors) and revisit the 0.9 list and bump all changes that can
wait
to 0.10 and then test the hell out of trunk.
Cheers
Jan
--
Re: I got a little carried away ...
Posted by Kerr Rainey <ke...@gmail.com>.
2009/3/5 Noah Slater <ns...@apache.org>:
> /me puts on his release manager hat
>
> Delay, cut 0.9 with the patches. Making a release is non-trivial.
I thought the point of trying to get a 0.9 release out was to have a
realse of all the stuff that's been sitting in trunk for a while?
Shouldn't we just get it out. It doesn't mean that we then need to do
another release in 2 weeks. There will always be new work that is 2
weeks away that could push back a release. At some stage something
isn't going to make it and it will go into the next cut.
But, you know, it's not like I'm the one that has to do any work. I
just think it would be good to get a release out.
--
Kerr
Re: I got a little carried away ...
Posted by Noah Slater <ns...@apache.org>.
On Thu, Mar 05, 2009 at 12:58:17PM +0100, Jan Lehnardt wrote:
>> Postpone by about 2 weeks, depending on when those changes land in
>> trunk?
>
> Fine with me. An alternative: Cut 0.9 now and merge the big patches for
> 0.10.
/me puts on his release manager hat
Delay, cut 0.9 with the patches. Making a release is non-trivial.
--
Noah Slater, http://tumbolia.org/nslater
Re: I got a little carried away ...
Posted by Kerr Rainey <ke...@gmail.com>.
2009/3/5 Jan Lehnardt <ja...@apache.org>:
> On 5 Mar 2009, at 13:04, Kerr Rainey wrote:
>> 2009/3/5 Jan Lehnardt <ja...@apache.org>:
>>>
>>> Fine with me. An alternative: Cut 0.9 now and merge the big patches for
>>> 0.10.
>>
>> +1
>
> + 1 to postponing or cutting now?
Sorry, I thought I'd trimmed enough to be clear. I'll try again.
> Cut 0.9 now and merge the big patches for 0.10.
+1
--
Kerr
Re: I got a little carried away ...
Posted by Jan Lehnardt <ja...@apache.org>.
On 5 Mar 2009, at 13:04, Kerr Rainey wrote:
> 2009/3/5 Jan Lehnardt <ja...@apache.org>:
>> Fine with me. An alternative: Cut 0.9 now and merge the big patches
>> for
>> 0.10.
>
> +1
+ 1 to postponing or cutting now?
Cheers
Jan
--
Re: I got a little carried away ...
Posted by Kerr Rainey <ke...@gmail.com>.
2009/3/5 Jan Lehnardt <ja...@apache.org>:
> Fine with me. An alternative: Cut 0.9 now and merge the big patches for
> 0.10.
+1
--
Kerr
Re: I got a little carried away ...
Posted by Jan Lehnardt <ja...@apache.org>.
On 5 Mar 2009, at 11:49, Christopher Lenz wrote:
> On 05.03.2009, at 07:19, Chris Anderson wrote:
>> We've now got three outstanding bigish patches that are candidates
>> for 0.9.0:
>>
>> Adam's OTP replication
>> Damien's rep_security
>> My design-doc resources
>>
>> merge party!
>
> If those are supposed to make it into 0.9.0, there's no way we can
> get the release out by the end of this week (which is pretty much
> today or tomorrow). Changes of this scale require review and
> testing, and ideally documentation. Just saying :)
My thoughts exactly.
> Postpone by about 2 weeks, depending on when those changes land in
> trunk?
Fine with me. An alternative: Cut 0.9 now and merge the big patches
for 0.10.
+0 on either.
Cheers
Jan
--
Re: I got a little carried away ...
Posted by Chris Anderson <jc...@apache.org>.
On Thu, Mar 5, 2009 at 9:18 AM, Chris Anderson <jc...@apache.org> wrote:
> On Thu, Mar 5, 2009 at 2:49 AM, Christopher Lenz <cm...@gmx.de> wrote:
>>
>> If those are supposed to make it into 0.9.0, there's no way we can get the
>> release out by the end of this week (which is pretty much today or
>> tomorrow). Changes of this scale require review and testing, and ideally
>> documentation. Just saying :)
>>
>> Postpone by about 2 weeks, depending on when those changes land in trunk?
>>
>
> I'd say postpone. Adam's replication patches don't change the API, so
> they aren't an issue for client libraries, but both moving the _view
> endpoint to within the design doc path, and Damien's changes to the
> behavior of bulk_docs, will effect client libraries.
>
> If we can avoid making a release with APIs that are already
> deprecated, I think it's worth the wait. Alternatively, we could wait
> on Damien's branch (as it still requires some testing, and changes
> CouchDB in a more fundamental way, but one that has less impact on
> client libraries) but I'd be against doing a release without the
> design doc resources patch. We've never had a release with show/list,
> so doing one with a deprecated API places an unnecessary burden on
> client and application developers, who will probably have to support
> 0.9.0 for a while.
>
That said, the design doc resources patch (COUCHDB-280) is solid and
tested and ready for trunk. I'm ready to commit it once I'm sure there
aren't any -1's out there.
--
Chris Anderson
http://jchris.mfdz.com
Re: I got a little carried away ...
Posted by Chris Anderson <jc...@apache.org>.
On Thu, Mar 5, 2009 at 2:49 AM, Christopher Lenz <cm...@gmx.de> wrote:
>
> If those are supposed to make it into 0.9.0, there's no way we can get the
> release out by the end of this week (which is pretty much today or
> tomorrow). Changes of this scale require review and testing, and ideally
> documentation. Just saying :)
>
> Postpone by about 2 weeks, depending on when those changes land in trunk?
>
I'd say postpone. Adam's replication patches don't change the API, so
they aren't an issue for client libraries, but both moving the _view
endpoint to within the design doc path, and Damien's changes to the
behavior of bulk_docs, will effect client libraries.
If we can avoid making a release with APIs that are already
deprecated, I think it's worth the wait. Alternatively, we could wait
on Damien's branch (as it still requires some testing, and changes
CouchDB in a more fundamental way, but one that has less impact on
client libraries) but I'd be against doing a release without the
design doc resources patch. We've never had a release with show/list,
so doing one with a deprecated API places an unnecessary burden on
client and application developers, who will probably have to support
0.9.0 for a while.
Chris
--
Chris Anderson
http://jchris.mfdz.com
Re: I got a little carried away ...
Posted by Christopher Lenz <cm...@gmx.de>.
On 05.03.2009, at 07:19, Chris Anderson wrote:
> We've now got three outstanding bigish patches that are candidates
> for 0.9.0:
>
> Adam's OTP replication
> Damien's rep_security
> My design-doc resources
>
> merge party!
If those are supposed to make it into 0.9.0, there's no way we can get
the release out by the end of this week (which is pretty much today or
tomorrow). Changes of this scale require review and testing, and
ideally documentation. Just saying :)
Postpone by about 2 weeks, depending on when those changes land in
trunk?
Cheers,
--
Christopher Lenz
cmlenz at gmx.de
http://www.cmlenz.net/
Re: I got a little carried away ...
Posted by Chris Anderson <jc...@apache.org>.
On Wed, Mar 4, 2009 at 6:03 PM, Chris Anderson <jc...@apache.org> wrote:
>
> I'm working on bringing Damien's rep_security branch up to date with
> trunk, this should help with the merging. Hopefully I'll be able to
> check in those changes tonight or tomorrow.
>
This has been checked in so the rep_security branch is up to date with
trunk as of r750305
We've now got three outstanding bigish patches that are candidates for 0.9.0:
Adam's OTP replication
Damien's rep_security
My design-doc resources
merge party!
--
Chris Anderson
http://jchris.mfdz.com
Re: I got a little carried away ...
Posted by Chris Anderson <jc...@apache.org>.
On Wed, Mar 4, 2009 at 5:34 PM, Adam Kocoloski <ad...@gmail.com> wrote:
> I was working through some replication tickets this week and thinking more
> and more that the replicator could benefit from being restructured along OTP
> principles. So, I went ahead and did it. Here's the structure I've worked
> out so far:
>
> http://github.com/kocolosk/couchdb/tree/otpify-replication
>
This is great news. I'll admit the replicator internals have never
been my area of expertise, but your description makes it sound like
the right direction. Robust is more important than fast at this stage
of the game, for sure.
I'm working on bringing Damien's rep_security branch up to date with
trunk, this should help with the merging. Hopefully I'll be able to
check in those changes tonight or tomorrow.
Chris
--
Chris Anderson
http://jchris.mfdz.com