You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Benoit Chesneau <bc...@gmail.com> on 2010/10/13 23:23:32 UTC
Using rebar to install couchdb
In an attempt to start some merging with cloudant I would like to
start by using rebar in our install process.
Like i see it, we could continue to use autotools to create the
rebar.config files and other templates an then rebar for the final
build and dependencies management. This changes as noticed by @davisp
also imply we make our tree a little more OTP compliant. I would like
to start this work asap.
Thoughts ?
- benoit
Re: Using rebar to install couchdb
Posted by Benoit Chesneau <bc...@gmail.com>.
On Thu, Oct 14, 2010 at 7:40 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> node,
>
s/node/note .
Re: Using rebar to install couchdb
Posted by Benoit Chesneau <bc...@gmail.com>.
On Thu, Oct 14, 2010 at 7:25 AM, Peter Lemenkov <le...@gmail.com> wrote:
> 2010/10/14 Benoit Chesneau <bc...@gmail.com>:
> Regarding switch to rebar - I don't see any significant benefits.
> Compiling *.erl files into *.beam and properly install them is not a
> rocket science and requires only few lines of shell scripting. At
> least on Linux. On the other hand I don't see any issues too (although
> I don't care about other OS than Linux)
>
rebar is very helpful to make erlang releases (using reltools) ready
for big deployments in production :
http://bitbucket.org/basho/rebar/wiki/ReleaseHandling
Our current build doesn't allow me to do that easily. Also on side
node, having CouchDB respecting more OTP conventions would be good
too.
- benoît
Re: Using rebar to install couchdb
Posted by Randall Leeds <ra...@gmail.com>.
I also didn't mean to imply that I object on those grounds. I just
wanted to bring it to everyone's attention, in case someone felt
strongly about it. I actually really like the autoconf + rebar
proposal.
-Randall
On Wed, Oct 13, 2010 at 22:29, Peter Lemenkov <le...@gmail.com> wrote:
> 2010/10/14 Peter Lemenkov <le...@gmail.com>:
>> 2010/10/14 Benoit Chesneau <bc...@gmail.com>:
>
>> I don't fully understand why people still believe that minimum
>> requirement for CouchDB is R12B5, because last released ver. 1.0.1
>> requires 5 additional patches for mochiweb and 2 patches for couchdb
>> itself just for passing self-tests.
>
> Ah, sorry - disregard my note about mochiweb. CouchDB bundles very old
> version of this library, which seems to be compatible with R12B. Also
> only one patch is required for CouchDB.
>
> --
> With best regards, Peter Lemenkov.
>
Re: Using rebar to install couchdb
Posted by Peter Lemenkov <le...@gmail.com>.
2010/10/14 Peter Lemenkov <le...@gmail.com>:
> 2010/10/14 Benoit Chesneau <bc...@gmail.com>:
> I don't fully understand why people still believe that minimum
> requirement for CouchDB is R12B5, because last released ver. 1.0.1
> requires 5 additional patches for mochiweb and 2 patches for couchdb
> itself just for passing self-tests.
Ah, sorry - disregard my note about mochiweb. CouchDB bundles very old
version of this library, which seems to be compatible with R12B. Also
only one patch is required for CouchDB.
--
With best regards, Peter Lemenkov.
Re: Using rebar to install couchdb
Posted by Peter Lemenkov <le...@gmail.com>.
2010/10/14 Benoit Chesneau <bc...@gmail.com>:
>>> The only concern I have is that I believe we need R13 for rebar and
>>> right now Couch only requires a minumum of R12B5, no? So it means
>>> dumping the dependency requirements.
> lapsus? ;) Most of the distributions are using erlang > r13b03, (even
> openbsd!), maybe not old centos, but I think it's ok to do that
> today.
I don't fully understand why people still believe that minimum
requirement for CouchDB is R12B5, because last released ver. 1.0.1
requires 5 additional patches for mochiweb and 2 patches for couchdb
itself just for passing self-tests.
Regarding switch to rebar - I don't see any significant benefits.
Compiling *.erl files into *.beam and properly install them is not a
rocket science and requires only few lines of shell scripting. At
least on Linux. On the other hand I don't see any issues too (although
I don't care about other OS than Linux)
--
With best regards, Peter Lemenkov.
Re: Using rebar to install couchdb
Posted by Benoit Chesneau <bc...@gmail.com>.
On Thu, Oct 14, 2010 at 12:01 AM, Randall Leeds <ra...@gmail.com> wrote:
> Errr... *B*umping the dependency requirements.
>
> On Wed, Oct 13, 2010 at 15:01, Randall Leeds <ra...@gmail.com> wrote:
>> The only concern I have is that I believe we need R13 for rebar and
>> right now Couch only requires a minumum of R12B5, no? So it means
>> dumping the dependency requirements.
>>
lapsus? ;) Most of the distributions are using erlang > r13b03, (even
openbsd!), maybe not old centos, but I think it's ok to do that
today. We will need to do it later. Also we need to bump requirements
for windows version anyway (r14b) .
- benoît
Re: Using rebar to install couchdb
Posted by Randall Leeds <ra...@gmail.com>.
Errr... *B*umping the dependency requirements.
On Wed, Oct 13, 2010 at 15:01, Randall Leeds <ra...@gmail.com> wrote:
> The only concern I have is that I believe we need R13 for rebar and
> right now Couch only requires a minumum of R12B5, no? So it means
> dumping the dependency requirements.
>
> On Wed, Oct 13, 2010 at 14:23, Benoit Chesneau <bc...@gmail.com> wrote:
>> In an attempt to start some merging with cloudant I would like to
>> start by using rebar in our install process.
>>
>> Like i see it, we could continue to use autotools to create the
>> rebar.config files and other templates an then rebar for the final
>> build and dependencies management. This changes as noticed by @davisp
>> also imply we make our tree a little more OTP compliant. I would like
>> to start this work asap.
>>
>> Thoughts ?
>>
>> - benoit
>>
>
Re: Using rebar to install couchdb
Posted by Randall Leeds <ra...@gmail.com>.
The only concern I have is that I believe we need R13 for rebar and
right now Couch only requires a minumum of R12B5, no? So it means
dumping the dependency requirements.
On Wed, Oct 13, 2010 at 14:23, Benoit Chesneau <bc...@gmail.com> wrote:
> In an attempt to start some merging with cloudant I would like to
> start by using rebar in our install process.
>
> Like i see it, we could continue to use autotools to create the
> rebar.config files and other templates an then rebar for the final
> build and dependencies management. This changes as noticed by @davisp
> also imply we make our tree a little more OTP compliant. I would like
> to start this work asap.
>
> Thoughts ?
>
> - benoit
>
Re: Using rebar to install couchdb
Posted by Joe Williams <jo...@joetify.com>.
On Oct 15, 2010, at 3:35 PM, Paul Davis wrote:
> On Fri, Oct 15, 2010 at 4:16 PM, Joe Williams <jo...@joetify.com> wrote:
>>
>> On Oct 13, 2010, at 4:23 PM, Benoit Chesneau wrote:
>>
>>> In an attempt to start some merging with cloudant I would like to
>>> start by using rebar in our install process.
>>>
>>> Like i see it, we could continue to use autotools to create the
>>> rebar.config files and other templates an then rebar for the final
>>> build and dependencies management. This changes as noticed by @davisp
>>> also imply we make our tree a little more OTP compliant. I would like
>>> to start this work asap.
>>>
>>> Thoughts ?
>>>
>>> - benoit
>>
>>
>> If it's any help I had this branch sitting on my local of a recent attempt at making CouchDB 1.0.1 use rebar similar to what we did with BigCouch. It seems to work well.
>>
>> http://github.com/joewilliams/couchdb/tree/rebar_1.0.1
>>
>>
>> -Joe
>>
>>
>>
>> Name: Joseph A. Williams
>> Email: joe@joetify.com
>> Blog: http://www.joeandmotorboat.com/
>> Twitter: http://twitter.com/williamsjoe
>>
>>
>
> Cool! This'll help getting the rebar configuration going.
>
> For those of you watching in TV land, this change is obviously quite a
> bit more extreme than the proposed changes. Just wanted to make a note
> so I don't have to explain that we're not going to be deleting things
> like the AUTHORS/THANKS/NOTICE files.
>
> Paul Davis
Thanks for the note Paul, this branch wasn't initially intended for consumption but I think it's certainly a good enough PoC. Also, I just added the LICENSE file and friends to keep things kosher.
-Joe
Name: Joseph A. Williams
Email: joe@joetify.com
Blog: http://www.joeandmotorboat.com/
Twitter: http://twitter.com/williamsjoe
Re: Using rebar to install couchdb
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Oct 15, 2010 at 4:16 PM, Joe Williams <jo...@joetify.com> wrote:
>
> On Oct 13, 2010, at 4:23 PM, Benoit Chesneau wrote:
>
>> In an attempt to start some merging with cloudant I would like to
>> start by using rebar in our install process.
>>
>> Like i see it, we could continue to use autotools to create the
>> rebar.config files and other templates an then rebar for the final
>> build and dependencies management. This changes as noticed by @davisp
>> also imply we make our tree a little more OTP compliant. I would like
>> to start this work asap.
>>
>> Thoughts ?
>>
>> - benoit
>
>
> If it's any help I had this branch sitting on my local of a recent attempt at making CouchDB 1.0.1 use rebar similar to what we did with BigCouch. It seems to work well.
>
> http://github.com/joewilliams/couchdb/tree/rebar_1.0.1
>
>
> -Joe
>
>
>
> Name: Joseph A. Williams
> Email: joe@joetify.com
> Blog: http://www.joeandmotorboat.com/
> Twitter: http://twitter.com/williamsjoe
>
>
Cool! This'll help getting the rebar configuration going.
For those of you watching in TV land, this change is obviously quite a
bit more extreme than the proposed changes. Just wanted to make a note
so I don't have to explain that we're not going to be deleting things
like the AUTHORS/THANKS/NOTICE files.
Paul Davis
Re: Using rebar to install couchdb
Posted by Benoit Chesneau <bc...@gmail.com>.
On Fri, Oct 15, 2010 at 10:16 PM, Joe Williams <jo...@joetify.com> wrote:
>
> On Oct 13, 2010, at 4:23 PM, Benoit Chesneau wrote:
>
>> In an attempt to start some merging with cloudant I would like to
>> start by using rebar in our install process.
>>
>> Like i see it, we could continue to use autotools to create the
>> rebar.config files and other templates an then rebar for the final
>> build and dependencies management. This changes as noticed by @davisp
>> also imply we make our tree a little more OTP compliant. I would like
>> to start this work asap.
>>
>> Thoughts ?
>>
>> - benoit
>
>
> If it's any help I had this branch sitting on my local of a recent attempt at making CouchDB 1.0.1 use rebar similar to what we did with BigCouch. It seems to work well.
>
> http://github.com/joewilliams/couchdb/tree/rebar_1.0.1
>
>
> -Joe
>
>
Pretty cool :) now need to figure how to add back configure and
deviltools friends. Also I think we want to do s/apps/src ?
- benoît
Re: Using rebar to install couchdb
Posted by Joe Williams <jo...@joetify.com>.
On Oct 13, 2010, at 4:23 PM, Benoit Chesneau wrote:
> In an attempt to start some merging with cloudant I would like to
> start by using rebar in our install process.
>
> Like i see it, we could continue to use autotools to create the
> rebar.config files and other templates an then rebar for the final
> build and dependencies management. This changes as noticed by @davisp
> also imply we make our tree a little more OTP compliant. I would like
> to start this work asap.
>
> Thoughts ?
>
> - benoit
If it's any help I had this branch sitting on my local of a recent attempt at making CouchDB 1.0.1 use rebar similar to what we did with BigCouch. It seems to work well.
http://github.com/joewilliams/couchdb/tree/rebar_1.0.1
-Joe
Name: Joseph A. Williams
Email: joe@joetify.com
Blog: http://www.joeandmotorboat.com/
Twitter: http://twitter.com/williamsjoe
Re: Using rebar to install couchdb
Posted by Benoit Chesneau <bc...@gmail.com>.
On Thu, Oct 14, 2010 at 8:54 PM, Paul Davis <pa...@gmail.com> wrote:
> So there's a couple issues at hand here which seem to be motivated by
> the desire to start using tools like rebar.
>
> Our current source tree is not compliant with some of the basic
> Erlang/OTP conventions. This is both bad technically and socially.
> Technically, it prevents us from easily integrating tools like rebar
> that would help advanced users with things like making Erlang reltools
> packages. Socially, it doesn't reflect well on us to members of the
> Erlang community that may have otherwise become contributors. All
> languages have a standard package layout and Erlang is no different.
>
> The current CouchDB Erlang app has grown considerably. There's been
> general consensus that we need to start splitting it up into smaller
> applications that encompass specific functionality. There's been a bit
> of effort in this direction, but its such a major change to source
> file location it needs to have a community consensus to really start
> working on seriously.
>
> I don't think we should focus directly on the issue of integrating
> rebar. It should definitely be a goal, but not at the cost of our
> current situation. Noah Slater has maintained an excellent build
> system for us as is shown by the number of people building CouchDB
> from source and the number of packages available. While I have argued
> with him on numerous occasions about details, I have come to the
> conclusion that it is not possible for him to be wrong. I personally
> attribute this to the fact that he's most likely an advanced robot
> from the future. That said, Noah has voiced concerns to various ideas
> and we should make sure that any of his concerns are fully addressed.
>
> We should attempt to make sure that any tool support doesn't morph
> into tool requirement. For instance, I think we should make sure that
> its possible to keep compiling CouchDB without rebar and not come to
> rely on it.
>
> While I'd be more than happy to start in on this and handle all of the
> build system refactoring to make this happen, I'm not going to start
> until there's a community consensus on what needs to be done. There
> are a couple paths that I could see us taking to make this happen. We
> could just make the current source tree be rebar compatible and figure
> out the build system to do the optional rebar build or we could also
> take this chance to split the source code into multiple applications.
> Personally, I'd prefer to take this opportunity to organize the code
> with multiple erlang apps.
>
> Too get the conversation rolling here's a first pass at a new app proposal:
>
> etap:
>
> Nick Gerakines now releases etap as a single .erl file that can be
> dropped into the test directory. This app should be removed in favor
> of that method.
>
> erlang-oauth:
>
> Should be renamed to just oauth. That erlang- prefix has bugged me
> fore entirely too long.
>
> mochiweb, ibrowse, oauth:
>
> Refactored to use standard src, include, ebin, priv directories to
> be OTP compliant. This results in directories like
>
> ./src/$APP/ebin
> ./src/$APP/incldue
> ./src/$APP/priv
> ./src/$APP/src
>
> couchdb:
>
> Each proposed app will be structured as described above. Proposed apps:
>
> couch_core: The core Erlang modules for storing docs and managing
> "internal infrastructure"
> couch_view: The view engine as well as the holder for managing OS processes.
> couch_rep: couch_rep*.erl
> couch_externals: couch_external*.erl
> couch_httpd: couch_http*.erl
>
> At the bottom of this email I made an initial pass through the
> ./src/couchdb tree to classify file by file into the described apps.
> There are also some minor warts in this split. Things like the core
> couchdb classes currently require calling into the couch_view app and
> what not. Given the "interesting" call graph for CouchDb internals, I
> think we should avoid addressing this and aim to minimize the actual
> change to source file contents during this move. After modules have
> been moved we can then start on teasing out the various cyclic
> dependencies as necessary.
>
> Also, on a last note, seeing as this is a major change to the source
> tree (and ideally 0 change to source code) it might be a good idea to
> start this work just after we release 1.1.0.
>
>
> What do you guys think?
>
> Paul Davis
>
>
I'm +1 for this plan :
1) split couchdb in different apps.
2) Put rebar to handle "the low level details for the OTP apps" (as said adam)
About 1 I asking myself if we shouldn't put rewrites and show in their
main apps "couchapp"
If we have a general consensus, I'm ready to start too.
- benoît
Re: Using rebar to install couchdb
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Oct 19, 2010 at 5:19 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Tue, Oct 19, 2010 at 9:29 PM, Paul Davis <pa...@gmail.com> wrote:
>
>> Just a quick note. I started actual (minimal) work on this last night.
>> After a bit of poking I'm starting to think that rebar isn't going to
>> support vpath builds without some contribution back to rebar. I'm not
>> against adding this but I also haven't gotten in touch with Dave Smith
>> yet to see what he thinks of adding it to begin with. So at the
>> moment, its a bit up in the air whether we can actually have a rebar
>> dependency for compiling CouchDB. I'm still very much focusing on at
>> least making it possible to use rebar via autotools, its just a matter
>> of whether that's via `GCC=rebar make` or some other mechanism.
>>
>
> Nit sure what do you mean exactly here. What we could do is a
> configure generating the rebar.config or if not possible directly via
> a make macro. My first idea was:
>
> configure -> generate different mae files + config, make using rebar to compile
>
The issue is vpath builds. To use rebar it needs to be able to place
compiled files into specific directories. Reading through the rebar
source, this does not look possible. I have a couple ideas to get
around it, but at this point I'm not sure I see it happening directly.
That's not too say I won't make it so that rebar can build in the
tree, but the official compiler can't be rebar until that's changed.
>
>> As I was looking through the rebar sources, I also realized that
>> there's a quite clean API for invoking the compiler. The actual build
>> loop of rebar is fairly slim, so I'm not entirely discounting writing
>> a slimmer rebar clone that we can hack on to our liking to make sure
>> it fits with autotools. Obviously this is the least desirable path
>> right now but I just wanted to mention its not out of the realm of
>> possibility.
>>
>> Also, I'm starting to fear how easy the SVN migration is going to be.
>> I haven't spent that much time going over the various svn behaviors,
>> but I wouldn't be suprised if we get to a point where this will have
>> to be a multi-commit episode over a weekend. If it happens that we'll
>> need to do this in multiple commits I'll probably setup a mirror of
>> the ASF repo so we can test the whole process before applying it to
>> trunk.
>>
>> I put up a very empty repo [1] last night where I'll be focusing work
>> on this. There's not a whole lot to it right now. If you want to help
>> out with testing or hacking on the scripts, send me a message on
>> github and I'll add you to the committers list for that repo.
>
> Do we really need a script for that ? Like I read it the script allows
> us to easily move in svn the files. Why not at first move like the
> original plan everything in its own folder (couchdb_core,
> couchdb_htpp, ...) and then use the current make file mecanism ? After
> that we could think to put in rebar.
>
I'm writing a script that will generate an svn snapshot that can be
committed. This way people can inspect the plan to move things without
having to stare at a 10K line patch. This move will be quite
complicated so the plan is to try and make the difference as palatable
as possible for people to digest.
> Wjhat do you think ?
>
> - benoît
>
HTH,
Paul Davis
Re: Using rebar to install couchdb
Posted by Benoit Chesneau <bc...@gmail.com>.
On Tue, Oct 19, 2010 at 9:29 PM, Paul Davis <pa...@gmail.com> wrote:
> Just a quick note. I started actual (minimal) work on this last night.
> After a bit of poking I'm starting to think that rebar isn't going to
> support vpath builds without some contribution back to rebar. I'm not
> against adding this but I also haven't gotten in touch with Dave Smith
> yet to see what he thinks of adding it to begin with. So at the
> moment, its a bit up in the air whether we can actually have a rebar
> dependency for compiling CouchDB. I'm still very much focusing on at
> least making it possible to use rebar via autotools, its just a matter
> of whether that's via `GCC=rebar make` or some other mechanism.
>
Nit sure what do you mean exactly here. What we could do is a
configure generating the rebar.config or if not possible directly via
a make macro. My first idea was:
configure -> generate different mae files + config, make using rebar to compile
> As I was looking through the rebar sources, I also realized that
> there's a quite clean API for invoking the compiler. The actual build
> loop of rebar is fairly slim, so I'm not entirely discounting writing
> a slimmer rebar clone that we can hack on to our liking to make sure
> it fits with autotools. Obviously this is the least desirable path
> right now but I just wanted to mention its not out of the realm of
> possibility.
>
> Also, I'm starting to fear how easy the SVN migration is going to be.
> I haven't spent that much time going over the various svn behaviors,
> but I wouldn't be suprised if we get to a point where this will have
> to be a multi-commit episode over a weekend. If it happens that we'll
> need to do this in multiple commits I'll probably setup a mirror of
> the ASF repo so we can test the whole process before applying it to
> trunk.
>
> I put up a very empty repo [1] last night where I'll be focusing work
> on this. There's not a whole lot to it right now. If you want to help
> out with testing or hacking on the scripts, send me a message on
> github and I'll add you to the committers list for that repo.
Do we really need a script for that ? Like I read it the script allows
us to easily move in svn the files. Why not at first move like the
original plan everything in its own folder (couchdb_core,
couchdb_htpp, ...) and then use the current make file mecanism ? After
that we could think to put in rebar.
Wjhat do you think ?
- benoît
Re: Using rebar to install couchdb
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Oct 19, 2010 at 3:29 PM, Paul Davis <pa...@gmail.com> wrote:
> On Thu, Oct 14, 2010 at 7:16 PM, Adam Kocoloski <ko...@apache.org> wrote:
>> On Oct 14, 2010, at 2:54 PM, Paul Davis wrote:
>>
>>> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>>>> In an attempt to start some merging with cloudant I would like to
>>>> start by using rebar in our install process.
>>>>
>>>> Like i see it, we could continue to use autotools to create the
>>>> rebar.config files and other templates an then rebar for the final
>>>> build and dependencies management. This changes as noticed by @davisp
>>>> also imply we make our tree a little more OTP compliant. I would like
>>>> to start this work asap.
>>>>
>>>> Thoughts ?
>>>>
>>>> - benoit
>>>>
>>>
>>> So there's a couple issues at hand here which seem to be motivated by
>>> the desire to start using tools like rebar.
>>>
>>> Our current source tree is not compliant with some of the basic
>>> Erlang/OTP conventions. This is both bad technically and socially.
>>> Technically, it prevents us from easily integrating tools like rebar
>>> that would help advanced users with things like making Erlang reltools
>>> packages. Socially, it doesn't reflect well on us to members of the
>>> Erlang community that may have otherwise become contributors. All
>>> languages have a standard package layout and Erlang is no different.
>>>
>>> The current CouchDB Erlang app has grown considerably. There's been
>>> general consensus that we need to start splitting it up into smaller
>>> applications that encompass specific functionality. There's been a bit
>>> of effort in this direction, but its such a major change to source
>>> file location it needs to have a community consensus to really start
>>> working on seriously.
>>>
>>> I don't think we should focus directly on the issue of integrating
>>> rebar. It should definitely be a goal, but not at the cost of our
>>> current situation. Noah Slater has maintained an excellent build
>>> system for us as is shown by the number of people building CouchDB
>>> from source and the number of packages available. While I have argued
>>> with him on numerous occasions about details, I have come to the
>>> conclusion that it is not possible for him to be wrong. I personally
>>> attribute this to the fact that he's most likely an advanced robot
>>> from the future. That said, Noah has voiced concerns to various ideas
>>> and we should make sure that any of his concerns are fully addressed.
>>>
>>> We should attempt to make sure that any tool support doesn't morph
>>> into tool requirement. For instance, I think we should make sure that
>>> its possible to keep compiling CouchDB without rebar and not come to
>>> rely on it.
>>
>> It should come as no surprise that I'm a big fan of rebar. I don't think we should avoid making it a requirement, because then we still have to do all the grunt work associated with keeping the pure autotools way of building our OTP applications working. Oh, and rebar supposedly does work with 12B-5, if we do still care about that. Assuming we can get a Windows-compatible rebar, I see no reason not to require it.
>>
>> I think Benoit is on the right track here - we keep autotools on top of everything, organizing the overall build and doing all the Apache-specific stuff. Rebar handles the low level details for the OTP apps.
>>
>> Definitely agree that the work should start in earnest shortly after 1.1.0. Best,
>>
>> Adam
>
> Just a quick note. I started actual (minimal) work on this last night.
> After a bit of poking I'm starting to think that rebar isn't going to
> support vpath builds without some contribution back to rebar. I'm not
> against adding this but I also haven't gotten in touch with Dave Smith
> yet to see what he thinks of adding it to begin with. So at the
> moment, its a bit up in the air whether we can actually have a rebar
> dependency for compiling CouchDB. I'm still very much focusing on at
> least making it possible to use rebar via autotools, its just a matter
> of whether that's via `GCC=rebar make` or some other mechanism.
>
> As I was looking through the rebar sources, I also realized that
> there's a quite clean API for invoking the compiler. The actual build
> loop of rebar is fairly slim, so I'm not entirely discounting writing
> a slimmer rebar clone that we can hack on to our liking to make sure
> it fits with autotools. Obviously this is the least desirable path
> right now but I just wanted to mention its not out of the realm of
> possibility.
>
> Also, I'm starting to fear how easy the SVN migration is going to be.
> I haven't spent that much time going over the various svn behaviors,
> but I wouldn't be suprised if we get to a point where this will have
> to be a multi-commit episode over a weekend. If it happens that we'll
> need to do this in multiple commits I'll probably setup a mirror of
> the ASF repo so we can test the whole process before applying it to
> trunk.
>
> I put up a very empty repo [1] last night where I'll be focusing work
> on this. There's not a whole lot to it right now. If you want to help
> out with testing or hacking on the scripts, send me a message on
> github and I'll add you to the committers list for that repo.
>
> Paul Davis
>
Whoopsie.
[1] http://github.com/davisp/couchdb-srcmv/
Re: Using rebar to install couchdb
Posted by Paul Davis <pa...@gmail.com>.
On Thu, Oct 14, 2010 at 7:16 PM, Adam Kocoloski <ko...@apache.org> wrote:
> On Oct 14, 2010, at 2:54 PM, Paul Davis wrote:
>
>> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>>> In an attempt to start some merging with cloudant I would like to
>>> start by using rebar in our install process.
>>>
>>> Like i see it, we could continue to use autotools to create the
>>> rebar.config files and other templates an then rebar for the final
>>> build and dependencies management. This changes as noticed by @davisp
>>> also imply we make our tree a little more OTP compliant. I would like
>>> to start this work asap.
>>>
>>> Thoughts ?
>>>
>>> - benoit
>>>
>>
>> So there's a couple issues at hand here which seem to be motivated by
>> the desire to start using tools like rebar.
>>
>> Our current source tree is not compliant with some of the basic
>> Erlang/OTP conventions. This is both bad technically and socially.
>> Technically, it prevents us from easily integrating tools like rebar
>> that would help advanced users with things like making Erlang reltools
>> packages. Socially, it doesn't reflect well on us to members of the
>> Erlang community that may have otherwise become contributors. All
>> languages have a standard package layout and Erlang is no different.
>>
>> The current CouchDB Erlang app has grown considerably. There's been
>> general consensus that we need to start splitting it up into smaller
>> applications that encompass specific functionality. There's been a bit
>> of effort in this direction, but its such a major change to source
>> file location it needs to have a community consensus to really start
>> working on seriously.
>>
>> I don't think we should focus directly on the issue of integrating
>> rebar. It should definitely be a goal, but not at the cost of our
>> current situation. Noah Slater has maintained an excellent build
>> system for us as is shown by the number of people building CouchDB
>> from source and the number of packages available. While I have argued
>> with him on numerous occasions about details, I have come to the
>> conclusion that it is not possible for him to be wrong. I personally
>> attribute this to the fact that he's most likely an advanced robot
>> from the future. That said, Noah has voiced concerns to various ideas
>> and we should make sure that any of his concerns are fully addressed.
>>
>> We should attempt to make sure that any tool support doesn't morph
>> into tool requirement. For instance, I think we should make sure that
>> its possible to keep compiling CouchDB without rebar and not come to
>> rely on it.
>
> It should come as no surprise that I'm a big fan of rebar. I don't think we should avoid making it a requirement, because then we still have to do all the grunt work associated with keeping the pure autotools way of building our OTP applications working. Oh, and rebar supposedly does work with 12B-5, if we do still care about that. Assuming we can get a Windows-compatible rebar, I see no reason not to require it.
>
> I think Benoit is on the right track here - we keep autotools on top of everything, organizing the overall build and doing all the Apache-specific stuff. Rebar handles the low level details for the OTP apps.
>
> Definitely agree that the work should start in earnest shortly after 1.1.0. Best,
>
> Adam
Just a quick note. I started actual (minimal) work on this last night.
After a bit of poking I'm starting to think that rebar isn't going to
support vpath builds without some contribution back to rebar. I'm not
against adding this but I also haven't gotten in touch with Dave Smith
yet to see what he thinks of adding it to begin with. So at the
moment, its a bit up in the air whether we can actually have a rebar
dependency for compiling CouchDB. I'm still very much focusing on at
least making it possible to use rebar via autotools, its just a matter
of whether that's via `GCC=rebar make` or some other mechanism.
As I was looking through the rebar sources, I also realized that
there's a quite clean API for invoking the compiler. The actual build
loop of rebar is fairly slim, so I'm not entirely discounting writing
a slimmer rebar clone that we can hack on to our liking to make sure
it fits with autotools. Obviously this is the least desirable path
right now but I just wanted to mention its not out of the realm of
possibility.
Also, I'm starting to fear how easy the SVN migration is going to be.
I haven't spent that much time going over the various svn behaviors,
but I wouldn't be suprised if we get to a point where this will have
to be a multi-commit episode over a weekend. If it happens that we'll
need to do this in multiple commits I'll probably setup a mirror of
the ASF repo so we can test the whole process before applying it to
trunk.
I put up a very empty repo [1] last night where I'll be focusing work
on this. There's not a whole lot to it right now. If you want to help
out with testing or hacking on the scripts, send me a message on
github and I'll add you to the committers list for that repo.
Paul Davis
Re: Using rebar to install couchdb
Posted by Adam Kocoloski <ko...@apache.org>.
On Oct 14, 2010, at 2:54 PM, Paul Davis wrote:
> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>> In an attempt to start some merging with cloudant I would like to
>> start by using rebar in our install process.
>>
>> Like i see it, we could continue to use autotools to create the
>> rebar.config files and other templates an then rebar for the final
>> build and dependencies management. This changes as noticed by @davisp
>> also imply we make our tree a little more OTP compliant. I would like
>> to start this work asap.
>>
>> Thoughts ?
>>
>> - benoit
>>
>
> So there's a couple issues at hand here which seem to be motivated by
> the desire to start using tools like rebar.
>
> Our current source tree is not compliant with some of the basic
> Erlang/OTP conventions. This is both bad technically and socially.
> Technically, it prevents us from easily integrating tools like rebar
> that would help advanced users with things like making Erlang reltools
> packages. Socially, it doesn't reflect well on us to members of the
> Erlang community that may have otherwise become contributors. All
> languages have a standard package layout and Erlang is no different.
>
> The current CouchDB Erlang app has grown considerably. There's been
> general consensus that we need to start splitting it up into smaller
> applications that encompass specific functionality. There's been a bit
> of effort in this direction, but its such a major change to source
> file location it needs to have a community consensus to really start
> working on seriously.
>
> I don't think we should focus directly on the issue of integrating
> rebar. It should definitely be a goal, but not at the cost of our
> current situation. Noah Slater has maintained an excellent build
> system for us as is shown by the number of people building CouchDB
> from source and the number of packages available. While I have argued
> with him on numerous occasions about details, I have come to the
> conclusion that it is not possible for him to be wrong. I personally
> attribute this to the fact that he's most likely an advanced robot
> from the future. That said, Noah has voiced concerns to various ideas
> and we should make sure that any of his concerns are fully addressed.
>
> We should attempt to make sure that any tool support doesn't morph
> into tool requirement. For instance, I think we should make sure that
> its possible to keep compiling CouchDB without rebar and not come to
> rely on it.
It should come as no surprise that I'm a big fan of rebar. I don't think we should avoid making it a requirement, because then we still have to do all the grunt work associated with keeping the pure autotools way of building our OTP applications working. Oh, and rebar supposedly does work with 12B-5, if we do still care about that. Assuming we can get a Windows-compatible rebar, I see no reason not to require it.
I think Benoit is on the right track here - we keep autotools on top of everything, organizing the overall build and doing all the Apache-specific stuff. Rebar handles the low level details for the OTP apps.
Definitely agree that the work should start in earnest shortly after 1.1.0. Best,
Adam
Re: Using rebar to install couchdb
Posted by Paul Davis <pa...@gmail.com>.
On Thu, Oct 14, 2010 at 6:52 PM, Robert Dionne
<di...@dionne-associates.com> wrote:
> +1 also
>
> I think the convention is
>
> ./apps/couch_core/ebin
> ./apps/couch_core/src
> ./apps/couch_core/include
> ./apps/couch_core/priv
> ./apps/couch_http/ebin
> ....
>
> rather than ./src/....
>
This is one of the places that Noah has expressed a view that all of
our compiled code should live under the ./src tree. I have seen the
./apps directory in Erlang projects, but I think its not nearly as
important as the src|ebin|include|priv structure inside the
application directories.
> I like the idea of still using the existing build, which is awesome, and have it feed into rebar so we can make use of reltools, etc, and templates for parameterizing the various .ini files
>
> doing it after the next release will be a good time to break everything
>
> On Oct 14, 2010, at 4:03 PM, Robert Newson wrote:
>
>> Paul,
>>
>> Brilliant writeup and proposal. I'd like to see all those things
>> happen pretty much as you said. Cleaning the cycles out will be much
>> easier once things are broken out in that style.
>>
>> +1
>>
>> B.
>>
>> On Thu, Oct 14, 2010 at 7:54 PM, Paul Davis <pa...@gmail.com> wrote:
>>> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>>>> In an attempt to start some merging with cloudant I would like to
>>>> start by using rebar in our install process.
>>>>
>>>> Like i see it, we could continue to use autotools to create the
>>>> rebar.config files and other templates an then rebar for the final
>>>> build and dependencies management. This changes as noticed by @davisp
>>>> also imply we make our tree a little more OTP compliant. I would like
>>>> to start this work asap.
>>>>
>>>> Thoughts ?
>>>>
>>>> - benoit
>>>>
>>>
>>> So there's a couple issues at hand here which seem to be motivated by
>>> the desire to start using tools like rebar.
>>>
>>> Our current source tree is not compliant with some of the basic
>>> Erlang/OTP conventions. This is both bad technically and socially.
>>> Technically, it prevents us from easily integrating tools like rebar
>>> that would help advanced users with things like making Erlang reltools
>>> packages. Socially, it doesn't reflect well on us to members of the
>>> Erlang community that may have otherwise become contributors. All
>>> languages have a standard package layout and Erlang is no different.
>>>
>>> The current CouchDB Erlang app has grown considerably. There's been
>>> general consensus that we need to start splitting it up into smaller
>>> applications that encompass specific functionality. There's been a bit
>>> of effort in this direction, but its such a major change to source
>>> file location it needs to have a community consensus to really start
>>> working on seriously.
>>>
>>> I don't think we should focus directly on the issue of integrating
>>> rebar. It should definitely be a goal, but not at the cost of our
>>> current situation. Noah Slater has maintained an excellent build
>>> system for us as is shown by the number of people building CouchDB
>>> from source and the number of packages available. While I have argued
>>> with him on numerous occasions about details, I have come to the
>>> conclusion that it is not possible for him to be wrong. I personally
>>> attribute this to the fact that he's most likely an advanced robot
>>> from the future. That said, Noah has voiced concerns to various ideas
>>> and we should make sure that any of his concerns are fully addressed.
>>>
>>> We should attempt to make sure that any tool support doesn't morph
>>> into tool requirement. For instance, I think we should make sure that
>>> its possible to keep compiling CouchDB without rebar and not come to
>>> rely on it.
>>>
>>> While I'd be more than happy to start in on this and handle all of the
>>> build system refactoring to make this happen, I'm not going to start
>>> until there's a community consensus on what needs to be done. There
>>> are a couple paths that I could see us taking to make this happen. We
>>> could just make the current source tree be rebar compatible and figure
>>> out the build system to do the optional rebar build or we could also
>>> take this chance to split the source code into multiple applications.
>>> Personally, I'd prefer to take this opportunity to organize the code
>>> with multiple erlang apps.
>>>
>>> Too get the conversation rolling here's a first pass at a new app proposal:
>>>
>>> etap:
>>>
>>> Nick Gerakines now releases etap as a single .erl file that can be
>>> dropped into the test directory. This app should be removed in favor
>>> of that method.
>>>
>>> erlang-oauth:
>>>
>>> Should be renamed to just oauth. That erlang- prefix has bugged me
>>> fore entirely too long.
>>>
>>> mochiweb, ibrowse, oauth:
>>>
>>> Refactored to use standard src, include, ebin, priv directories to
>>> be OTP compliant. This results in directories like
>>>
>>> ./src/$APP/ebin
>>> ./src/$APP/incldue
>>> ./src/$APP/priv
>>> ./src/$APP/src
>>>
>>> couchdb:
>>>
>>> Each proposed app will be structured as described above. Proposed apps:
>>>
>>> couch_core: The core Erlang modules for storing docs and managing
>>> "internal infrastructure"
>>> couch_view: The view engine as well as the holder for managing OS processes.
>>> couch_rep: couch_rep*.erl
>>> couch_externals: couch_external*.erl
>>> couch_httpd: couch_http*.erl
>>>
>>> At the bottom of this email I made an initial pass through the
>>> ./src/couchdb tree to classify file by file into the described apps.
>>> There are also some minor warts in this split. Things like the core
>>> couchdb classes currently require calling into the couch_view app and
>>> what not. Given the "interesting" call graph for CouchDb internals, I
>>> think we should avoid addressing this and aim to minimize the actual
>>> change to source file contents during this move. After modules have
>>> been moved we can then start on teasing out the various cyclic
>>> dependencies as necessary.
>>>
>>> Also, on a last note, seeing as this is a major change to the source
>>> tree (and ideally 0 change to source code) it might be a good idea to
>>> start this work just after we release 1.1.0.
>>>
>>>
>>> What do you guys think?
>>>
>>> Paul Davis
>>>
>>>
>>> List of files by per app:
>>>
>>> couch_core:
>>>
>>> ./src/couchdb/couch.erl
>>> ./src/couchdb/couch_app.erl
>>> ./src/couchdb/couch_auth_cache.erl
>>> ./src/couchdb/couch_btree.erl
>>> ./src/couchdb/couch_changes.erl
>>> ./src/couchdb/couch_config.erl
>>> ./src/couchdb/couch_config_writer.erl
>>> ./src/couchdb/couch_db.erl
>>> ./src/couchdb/couch_db.hrl
>>> ./src/couchdb/couch_db_update_notifier.erl
>>> ./src/couchdb/couch_db_update_notifier_sup.erl
>>> ./src/couchdb/couch_db_updater.erl
>>> ./src/couchdb/couch_doc.erl
>>> ./src/couchdb/couch_event_sup.erl
>>> ./src/couchdb/couch_file.erl
>>> ./src/couchdb/couch_key_tree.erl
>>> ./src/couchdb/couch_log.erl
>>> ./src/couchdb/couch_ref_counter.erl
>>> ./src/couchdb/couch_server.erl
>>> ./src/couchdb/couch_server_sup.erl
>>> ./src/couchdb/couch_stats_aggregator.erl
>>> ./src/couchdb/couch_stats_collector.erl
>>> ./src/couchdb/couch_stream.erl
>>> ./src/couchdb/couch_task_status.erl
>>> ./src/couchdb/couch_util.erl
>>> ./src/couchdb/couch_uuids.erl
>>> ./src/couchdb/priv/icu_driver/couch_icu_driver.c
>>> ./src/couchdb/priv/stat_descriptions.cfg.in
>>>
>>> couch_view:
>>>
>>> ./src/couchdb/couch_native_process.erl
>>> ./src/couchdb/couch_os_process.erl
>>> ./src/couchdb/couch_query_servers.erl
>>> ./src/couchdb/couch_view.erl
>>> ./src/couchdb/couch_view_compactor.erl
>>> ./src/couchdb/couch_view_group.erl
>>> ./src/couchdb/couch_view_updater.erl
>>> ./src/couchdb/couch_work_queue.erl
>>> ./src/couchdb/priv/couch_js/http.c
>>> ./src/couchdb/priv/couch_js/http.h
>>> ./src/couchdb/priv/couch_js/main.c
>>> ./src/couchdb/priv/couch_js/utf8.c
>>> ./src/couchdb/priv/couch_js/utf8.h
>>> ./src/couchdb/priv/spawnkillable/couchspawnkillable.sh
>>> ./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
>>>
>>> couch_rep:
>>>
>>> ./src/couchdb/couch_js_functions.hrl
>>> ./src/couchdb/couch_rep.erl
>>> ./src/couchdb/couch_rep_att.erl
>>> ./src/couchdb/couch_rep_changes_feed.erl
>>> ./src/couchdb/couch_rep_db_listener.erl
>>> ./src/couchdb/couch_rep_httpc.erl
>>> ./src/couchdb/couch_rep_missing_revs.erl
>>> ./src/couchdb/couch_rep_reader.erl
>>> ./src/couchdb/couch_rep_sup.erl
>>> ./src/couchdb/couch_rep_writer.erl
>>>
>>> couch_externals:
>>>
>>> ./src/couchdb/couch_external_manager.erl
>>> ./src/couchdb/couch_external_server.erl
>>>
>>> couch_httpd:
>>>
>>> ./src/couchdb/couch_httpd.erl
>>> ./src/couchdb/couch_httpd_auth.erl
>>> ./src/couchdb/couch_httpd_db.erl
>>> ./src/couchdb/couch_httpd_external.erl
>>> ./src/couchdb/couch_httpd_misc_handlers.erl
>>> ./src/couchdb/couch_httpd_oauth.erl
>>> ./src/couchdb/couch_httpd_rewrite.erl
>>> ./src/couchdb/couch_httpd_show.erl
>>> ./src/couchdb/couch_httpd_stats_handlers.erl
>>> ./src/couchdb/couch_httpd_vhost.erl
>>> ./src/couchdb/couch_httpd_view.erl
>>>
>
>
Re: Using rebar to install couchdb
Posted by Adam Kocoloski <ko...@apache.org>.
On Oct 14, 2010, at 7:16 PM, Paul Davis wrote:
> On Thu, Oct 14, 2010 at 7:01 PM, Brian Mitchell <bi...@gmail.com> wrote:
>> couch_* seems a little repetitive. Why not core/, http/, etc?
>>
>
> I think its important to keep couch in the app directory name. This
> will come into play if people pull in couchdb code into other
> packages. For instance, I think it'd make things confusing when people
> make packages like BigCouch.
Yes, it's repetitive, but since all Erlang modules share a flat namespace it's definitely the right approach IMO. Best,
Adam
Re: Using rebar to install couchdb
Posted by Paul Davis <pa...@gmail.com>.
On Thu, Oct 14, 2010 at 7:01 PM, Brian Mitchell <bi...@gmail.com> wrote:
> couch_* seems a little repetitive. Why not core/, http/, etc?
>
I think its important to keep couch in the app directory name. This
will come into play if people pull in couchdb code into other
packages. For instance, I think it'd make things confusing when people
make packages like BigCouch.
>
>
> On Oct 14, 2010, at 18:52, Robert Dionne <di...@dionne-associates.com> wrote:
>
>> +1 also
>>
>> I think the convention is
>>
>> ./apps/couch_core/ebin
>> ./apps/couch_core/src
>> ./apps/couch_core/include
>> ./apps/couch_core/priv
>> ./apps/couch_http/ebin
>> ....
>>
>> rather than ./src/....
>>
>> I like the idea of still using the existing build, which is awesome, and have it feed into rebar so we can make use of reltools, etc, and templates for parameterizing the various .ini files
>>
>> doing it after the next release will be a good time to break everything
>>
>> On Oct 14, 2010, at 4:03 PM, Robert Newson wrote:
>>
>>> Paul,
>>>
>>> Brilliant writeup and proposal. I'd like to see all those things
>>> happen pretty much as you said. Cleaning the cycles out will be much
>>> easier once things are broken out in that style.
>>>
>>> +1
>>>
>>> B.
>>>
>>> On Thu, Oct 14, 2010 at 7:54 PM, Paul Davis <pa...@gmail.com> wrote:
>>>> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>>>>> In an attempt to start some merging with cloudant I would like to
>>>>> start by using rebar in our install process.
>>>>>
>>>>> Like i see it, we could continue to use autotools to create the
>>>>> rebar.config files and other templates an then rebar for the final
>>>>> build and dependencies management. This changes as noticed by @davisp
>>>>> also imply we make our tree a little more OTP compliant. I would like
>>>>> to start this work asap.
>>>>>
>>>>> Thoughts ?
>>>>>
>>>>> - benoit
>>>>>
>>>>
>>>> So there's a couple issues at hand here which seem to be motivated by
>>>> the desire to start using tools like rebar.
>>>>
>>>> Our current source tree is not compliant with some of the basic
>>>> Erlang/OTP conventions. This is both bad technically and socially.
>>>> Technically, it prevents us from easily integrating tools like rebar
>>>> that would help advanced users with things like making Erlang reltools
>>>> packages. Socially, it doesn't reflect well on us to members of the
>>>> Erlang community that may have otherwise become contributors. All
>>>> languages have a standard package layout and Erlang is no different.
>>>>
>>>> The current CouchDB Erlang app has grown considerably. There's been
>>>> general consensus that we need to start splitting it up into smaller
>>>> applications that encompass specific functionality. There's been a bit
>>>> of effort in this direction, but its such a major change to source
>>>> file location it needs to have a community consensus to really start
>>>> working on seriously.
>>>>
>>>> I don't think we should focus directly on the issue of integrating
>>>> rebar. It should definitely be a goal, but not at the cost of our
>>>> current situation. Noah Slater has maintained an excellent build
>>>> system for us as is shown by the number of people building CouchDB
>>>> from source and the number of packages available. While I have argued
>>>> with him on numerous occasions about details, I have come to the
>>>> conclusion that it is not possible for him to be wrong. I personally
>>>> attribute this to the fact that he's most likely an advanced robot
>>>> from the future. That said, Noah has voiced concerns to various ideas
>>>> and we should make sure that any of his concerns are fully addressed.
>>>>
>>>> We should attempt to make sure that any tool support doesn't morph
>>>> into tool requirement. For instance, I think we should make sure that
>>>> its possible to keep compiling CouchDB without rebar and not come to
>>>> rely on it.
>>>>
>>>> While I'd be more than happy to start in on this and handle all of the
>>>> build system refactoring to make this happen, I'm not going to start
>>>> until there's a community consensus on what needs to be done. There
>>>> are a couple paths that I could see us taking to make this happen. We
>>>> could just make the current source tree be rebar compatible and figure
>>>> out the build system to do the optional rebar build or we could also
>>>> take this chance to split the source code into multiple applications.
>>>> Personally, I'd prefer to take this opportunity to organize the code
>>>> with multiple erlang apps.
>>>>
>>>> Too get the conversation rolling here's a first pass at a new app proposal:
>>>>
>>>> etap:
>>>>
>>>> Nick Gerakines now releases etap as a single .erl file that can be
>>>> dropped into the test directory. This app should be removed in favor
>>>> of that method.
>>>>
>>>> erlang-oauth:
>>>>
>>>> Should be renamed to just oauth. That erlang- prefix has bugged me
>>>> fore entirely too long.
>>>>
>>>> mochiweb, ibrowse, oauth:
>>>>
>>>> Refactored to use standard src, include, ebin, priv directories to
>>>> be OTP compliant. This results in directories like
>>>>
>>>> ./src/$APP/ebin
>>>> ./src/$APP/incldue
>>>> ./src/$APP/priv
>>>> ./src/$APP/src
>>>>
>>>> couchdb:
>>>>
>>>> Each proposed app will be structured as described above. Proposed apps:
>>>>
>>>> couch_core: The core Erlang modules for storing docs and managing
>>>> "internal infrastructure"
>>>> couch_view: The view engine as well as the holder for managing OS processes.
>>>> couch_rep: couch_rep*.erl
>>>> couch_externals: couch_external*.erl
>>>> couch_httpd: couch_http*.erl
>>>>
>>>> At the bottom of this email I made an initial pass through the
>>>> ./src/couchdb tree to classify file by file into the described apps.
>>>> There are also some minor warts in this split. Things like the core
>>>> couchdb classes currently require calling into the couch_view app and
>>>> what not. Given the "interesting" call graph for CouchDb internals, I
>>>> think we should avoid addressing this and aim to minimize the actual
>>>> change to source file contents during this move. After modules have
>>>> been moved we can then start on teasing out the various cyclic
>>>> dependencies as necessary.
>>>>
>>>> Also, on a last note, seeing as this is a major change to the source
>>>> tree (and ideally 0 change to source code) it might be a good idea to
>>>> start this work just after we release 1.1.0.
>>>>
>>>>
>>>> What do you guys think?
>>>>
>>>> Paul Davis
>>>>
>>>>
>>>> List of files by per app:
>>>>
>>>> couch_core:
>>>>
>>>> ./src/couchdb/couch.erl
>>>> ./src/couchdb/couch_app.erl
>>>> ./src/couchdb/couch_auth_cache.erl
>>>> ./src/couchdb/couch_btree.erl
>>>> ./src/couchdb/couch_changes.erl
>>>> ./src/couchdb/couch_config.erl
>>>> ./src/couchdb/couch_config_writer.erl
>>>> ./src/couchdb/couch_db.erl
>>>> ./src/couchdb/couch_db.hrl
>>>> ./src/couchdb/couch_db_update_notifier.erl
>>>> ./src/couchdb/couch_db_update_notifier_sup.erl
>>>> ./src/couchdb/couch_db_updater.erl
>>>> ./src/couchdb/couch_doc.erl
>>>> ./src/couchdb/couch_event_sup.erl
>>>> ./src/couchdb/couch_file.erl
>>>> ./src/couchdb/couch_key_tree.erl
>>>> ./src/couchdb/couch_log.erl
>>>> ./src/couchdb/couch_ref_counter.erl
>>>> ./src/couchdb/couch_server.erl
>>>> ./src/couchdb/couch_server_sup.erl
>>>> ./src/couchdb/couch_stats_aggregator.erl
>>>> ./src/couchdb/couch_stats_collector.erl
>>>> ./src/couchdb/couch_stream.erl
>>>> ./src/couchdb/couch_task_status.erl
>>>> ./src/couchdb/couch_util.erl
>>>> ./src/couchdb/couch_uuids.erl
>>>> ./src/couchdb/priv/icu_driver/couch_icu_driver.c
>>>> ./src/couchdb/priv/stat_descriptions.cfg.in
>>>>
>>>> couch_view:
>>>>
>>>> ./src/couchdb/couch_native_process.erl
>>>> ./src/couchdb/couch_os_process.erl
>>>> ./src/couchdb/couch_query_servers.erl
>>>> ./src/couchdb/couch_view.erl
>>>> ./src/couchdb/couch_view_compactor.erl
>>>> ./src/couchdb/couch_view_group.erl
>>>> ./src/couchdb/couch_view_updater.erl
>>>> ./src/couchdb/couch_work_queue.erl
>>>> ./src/couchdb/priv/couch_js/http.c
>>>> ./src/couchdb/priv/couch_js/http.h
>>>> ./src/couchdb/priv/couch_js/main.c
>>>> ./src/couchdb/priv/couch_js/utf8.c
>>>> ./src/couchdb/priv/couch_js/utf8.h
>>>> ./src/couchdb/priv/spawnkillable/couchspawnkillable.sh
>>>> ./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
>>>>
>>>> couch_rep:
>>>>
>>>> ./src/couchdb/couch_js_functions.hrl
>>>> ./src/couchdb/couch_rep.erl
>>>> ./src/couchdb/couch_rep_att.erl
>>>> ./src/couchdb/couch_rep_changes_feed.erl
>>>> ./src/couchdb/couch_rep_db_listener.erl
>>>> ./src/couchdb/couch_rep_httpc.erl
>>>> ./src/couchdb/couch_rep_missing_revs.erl
>>>> ./src/couchdb/couch_rep_reader.erl
>>>> ./src/couchdb/couch_rep_sup.erl
>>>> ./src/couchdb/couch_rep_writer.erl
>>>>
>>>> couch_externals:
>>>>
>>>> ./src/couchdb/couch_external_manager.erl
>>>> ./src/couchdb/couch_external_server.erl
>>>>
>>>> couch_httpd:
>>>>
>>>> ./src/couchdb/couch_httpd.erl
>>>> ./src/couchdb/couch_httpd_auth.erl
>>>> ./src/couchdb/couch_httpd_db.erl
>>>> ./src/couchdb/couch_httpd_external.erl
>>>> ./src/couchdb/couch_httpd_misc_handlers.erl
>>>> ./src/couchdb/couch_httpd_oauth.erl
>>>> ./src/couchdb/couch_httpd_rewrite.erl
>>>> ./src/couchdb/couch_httpd_show.erl
>>>> ./src/couchdb/couch_httpd_stats_handlers.erl
>>>> ./src/couchdb/couch_httpd_vhost.erl
>>>> ./src/couchdb/couch_httpd_view.erl
>>>>
>>
>
Re: Using rebar to install couchdb
Posted by Brian Mitchell <bi...@gmail.com>.
couch_* seems a little repetitive. Why not core/, http/, etc?
On Oct 14, 2010, at 18:52, Robert Dionne <di...@dionne-associates.com> wrote:
> +1 also
>
> I think the convention is
>
> ./apps/couch_core/ebin
> ./apps/couch_core/src
> ./apps/couch_core/include
> ./apps/couch_core/priv
> ./apps/couch_http/ebin
> ....
>
> rather than ./src/....
>
> I like the idea of still using the existing build, which is awesome, and have it feed into rebar so we can make use of reltools, etc, and templates for parameterizing the various .ini files
>
> doing it after the next release will be a good time to break everything
>
> On Oct 14, 2010, at 4:03 PM, Robert Newson wrote:
>
>> Paul,
>>
>> Brilliant writeup and proposal. I'd like to see all those things
>> happen pretty much as you said. Cleaning the cycles out will be much
>> easier once things are broken out in that style.
>>
>> +1
>>
>> B.
>>
>> On Thu, Oct 14, 2010 at 7:54 PM, Paul Davis <pa...@gmail.com> wrote:
>>> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>>>> In an attempt to start some merging with cloudant I would like to
>>>> start by using rebar in our install process.
>>>>
>>>> Like i see it, we could continue to use autotools to create the
>>>> rebar.config files and other templates an then rebar for the final
>>>> build and dependencies management. This changes as noticed by @davisp
>>>> also imply we make our tree a little more OTP compliant. I would like
>>>> to start this work asap.
>>>>
>>>> Thoughts ?
>>>>
>>>> - benoit
>>>>
>>>
>>> So there's a couple issues at hand here which seem to be motivated by
>>> the desire to start using tools like rebar.
>>>
>>> Our current source tree is not compliant with some of the basic
>>> Erlang/OTP conventions. This is both bad technically and socially.
>>> Technically, it prevents us from easily integrating tools like rebar
>>> that would help advanced users with things like making Erlang reltools
>>> packages. Socially, it doesn't reflect well on us to members of the
>>> Erlang community that may have otherwise become contributors. All
>>> languages have a standard package layout and Erlang is no different.
>>>
>>> The current CouchDB Erlang app has grown considerably. There's been
>>> general consensus that we need to start splitting it up into smaller
>>> applications that encompass specific functionality. There's been a bit
>>> of effort in this direction, but its such a major change to source
>>> file location it needs to have a community consensus to really start
>>> working on seriously.
>>>
>>> I don't think we should focus directly on the issue of integrating
>>> rebar. It should definitely be a goal, but not at the cost of our
>>> current situation. Noah Slater has maintained an excellent build
>>> system for us as is shown by the number of people building CouchDB
>>> from source and the number of packages available. While I have argued
>>> with him on numerous occasions about details, I have come to the
>>> conclusion that it is not possible for him to be wrong. I personally
>>> attribute this to the fact that he's most likely an advanced robot
>>> from the future. That said, Noah has voiced concerns to various ideas
>>> and we should make sure that any of his concerns are fully addressed.
>>>
>>> We should attempt to make sure that any tool support doesn't morph
>>> into tool requirement. For instance, I think we should make sure that
>>> its possible to keep compiling CouchDB without rebar and not come to
>>> rely on it.
>>>
>>> While I'd be more than happy to start in on this and handle all of the
>>> build system refactoring to make this happen, I'm not going to start
>>> until there's a community consensus on what needs to be done. There
>>> are a couple paths that I could see us taking to make this happen. We
>>> could just make the current source tree be rebar compatible and figure
>>> out the build system to do the optional rebar build or we could also
>>> take this chance to split the source code into multiple applications.
>>> Personally, I'd prefer to take this opportunity to organize the code
>>> with multiple erlang apps.
>>>
>>> Too get the conversation rolling here's a first pass at a new app proposal:
>>>
>>> etap:
>>>
>>> Nick Gerakines now releases etap as a single .erl file that can be
>>> dropped into the test directory. This app should be removed in favor
>>> of that method.
>>>
>>> erlang-oauth:
>>>
>>> Should be renamed to just oauth. That erlang- prefix has bugged me
>>> fore entirely too long.
>>>
>>> mochiweb, ibrowse, oauth:
>>>
>>> Refactored to use standard src, include, ebin, priv directories to
>>> be OTP compliant. This results in directories like
>>>
>>> ./src/$APP/ebin
>>> ./src/$APP/incldue
>>> ./src/$APP/priv
>>> ./src/$APP/src
>>>
>>> couchdb:
>>>
>>> Each proposed app will be structured as described above. Proposed apps:
>>>
>>> couch_core: The core Erlang modules for storing docs and managing
>>> "internal infrastructure"
>>> couch_view: The view engine as well as the holder for managing OS processes.
>>> couch_rep: couch_rep*.erl
>>> couch_externals: couch_external*.erl
>>> couch_httpd: couch_http*.erl
>>>
>>> At the bottom of this email I made an initial pass through the
>>> ./src/couchdb tree to classify file by file into the described apps.
>>> There are also some minor warts in this split. Things like the core
>>> couchdb classes currently require calling into the couch_view app and
>>> what not. Given the "interesting" call graph for CouchDb internals, I
>>> think we should avoid addressing this and aim to minimize the actual
>>> change to source file contents during this move. After modules have
>>> been moved we can then start on teasing out the various cyclic
>>> dependencies as necessary.
>>>
>>> Also, on a last note, seeing as this is a major change to the source
>>> tree (and ideally 0 change to source code) it might be a good idea to
>>> start this work just after we release 1.1.0.
>>>
>>>
>>> What do you guys think?
>>>
>>> Paul Davis
>>>
>>>
>>> List of files by per app:
>>>
>>> couch_core:
>>>
>>> ./src/couchdb/couch.erl
>>> ./src/couchdb/couch_app.erl
>>> ./src/couchdb/couch_auth_cache.erl
>>> ./src/couchdb/couch_btree.erl
>>> ./src/couchdb/couch_changes.erl
>>> ./src/couchdb/couch_config.erl
>>> ./src/couchdb/couch_config_writer.erl
>>> ./src/couchdb/couch_db.erl
>>> ./src/couchdb/couch_db.hrl
>>> ./src/couchdb/couch_db_update_notifier.erl
>>> ./src/couchdb/couch_db_update_notifier_sup.erl
>>> ./src/couchdb/couch_db_updater.erl
>>> ./src/couchdb/couch_doc.erl
>>> ./src/couchdb/couch_event_sup.erl
>>> ./src/couchdb/couch_file.erl
>>> ./src/couchdb/couch_key_tree.erl
>>> ./src/couchdb/couch_log.erl
>>> ./src/couchdb/couch_ref_counter.erl
>>> ./src/couchdb/couch_server.erl
>>> ./src/couchdb/couch_server_sup.erl
>>> ./src/couchdb/couch_stats_aggregator.erl
>>> ./src/couchdb/couch_stats_collector.erl
>>> ./src/couchdb/couch_stream.erl
>>> ./src/couchdb/couch_task_status.erl
>>> ./src/couchdb/couch_util.erl
>>> ./src/couchdb/couch_uuids.erl
>>> ./src/couchdb/priv/icu_driver/couch_icu_driver.c
>>> ./src/couchdb/priv/stat_descriptions.cfg.in
>>>
>>> couch_view:
>>>
>>> ./src/couchdb/couch_native_process.erl
>>> ./src/couchdb/couch_os_process.erl
>>> ./src/couchdb/couch_query_servers.erl
>>> ./src/couchdb/couch_view.erl
>>> ./src/couchdb/couch_view_compactor.erl
>>> ./src/couchdb/couch_view_group.erl
>>> ./src/couchdb/couch_view_updater.erl
>>> ./src/couchdb/couch_work_queue.erl
>>> ./src/couchdb/priv/couch_js/http.c
>>> ./src/couchdb/priv/couch_js/http.h
>>> ./src/couchdb/priv/couch_js/main.c
>>> ./src/couchdb/priv/couch_js/utf8.c
>>> ./src/couchdb/priv/couch_js/utf8.h
>>> ./src/couchdb/priv/spawnkillable/couchspawnkillable.sh
>>> ./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
>>>
>>> couch_rep:
>>>
>>> ./src/couchdb/couch_js_functions.hrl
>>> ./src/couchdb/couch_rep.erl
>>> ./src/couchdb/couch_rep_att.erl
>>> ./src/couchdb/couch_rep_changes_feed.erl
>>> ./src/couchdb/couch_rep_db_listener.erl
>>> ./src/couchdb/couch_rep_httpc.erl
>>> ./src/couchdb/couch_rep_missing_revs.erl
>>> ./src/couchdb/couch_rep_reader.erl
>>> ./src/couchdb/couch_rep_sup.erl
>>> ./src/couchdb/couch_rep_writer.erl
>>>
>>> couch_externals:
>>>
>>> ./src/couchdb/couch_external_manager.erl
>>> ./src/couchdb/couch_external_server.erl
>>>
>>> couch_httpd:
>>>
>>> ./src/couchdb/couch_httpd.erl
>>> ./src/couchdb/couch_httpd_auth.erl
>>> ./src/couchdb/couch_httpd_db.erl
>>> ./src/couchdb/couch_httpd_external.erl
>>> ./src/couchdb/couch_httpd_misc_handlers.erl
>>> ./src/couchdb/couch_httpd_oauth.erl
>>> ./src/couchdb/couch_httpd_rewrite.erl
>>> ./src/couchdb/couch_httpd_show.erl
>>> ./src/couchdb/couch_httpd_stats_handlers.erl
>>> ./src/couchdb/couch_httpd_vhost.erl
>>> ./src/couchdb/couch_httpd_view.erl
>>>
>
Re: Using rebar to install couchdb
Posted by Robert Dionne <di...@dionne-associates.com>.
+1 also
I think the convention is
./apps/couch_core/ebin
./apps/couch_core/src
./apps/couch_core/include
./apps/couch_core/priv
./apps/couch_http/ebin
....
rather than ./src/....
I like the idea of still using the existing build, which is awesome, and have it feed into rebar so we can make use of reltools, etc, and templates for parameterizing the various .ini files
doing it after the next release will be a good time to break everything
On Oct 14, 2010, at 4:03 PM, Robert Newson wrote:
> Paul,
>
> Brilliant writeup and proposal. I'd like to see all those things
> happen pretty much as you said. Cleaning the cycles out will be much
> easier once things are broken out in that style.
>
> +1
>
> B.
>
> On Thu, Oct 14, 2010 at 7:54 PM, Paul Davis <pa...@gmail.com> wrote:
>> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>>> In an attempt to start some merging with cloudant I would like to
>>> start by using rebar in our install process.
>>>
>>> Like i see it, we could continue to use autotools to create the
>>> rebar.config files and other templates an then rebar for the final
>>> build and dependencies management. This changes as noticed by @davisp
>>> also imply we make our tree a little more OTP compliant. I would like
>>> to start this work asap.
>>>
>>> Thoughts ?
>>>
>>> - benoit
>>>
>>
>> So there's a couple issues at hand here which seem to be motivated by
>> the desire to start using tools like rebar.
>>
>> Our current source tree is not compliant with some of the basic
>> Erlang/OTP conventions. This is both bad technically and socially.
>> Technically, it prevents us from easily integrating tools like rebar
>> that would help advanced users with things like making Erlang reltools
>> packages. Socially, it doesn't reflect well on us to members of the
>> Erlang community that may have otherwise become contributors. All
>> languages have a standard package layout and Erlang is no different.
>>
>> The current CouchDB Erlang app has grown considerably. There's been
>> general consensus that we need to start splitting it up into smaller
>> applications that encompass specific functionality. There's been a bit
>> of effort in this direction, but its such a major change to source
>> file location it needs to have a community consensus to really start
>> working on seriously.
>>
>> I don't think we should focus directly on the issue of integrating
>> rebar. It should definitely be a goal, but not at the cost of our
>> current situation. Noah Slater has maintained an excellent build
>> system for us as is shown by the number of people building CouchDB
>> from source and the number of packages available. While I have argued
>> with him on numerous occasions about details, I have come to the
>> conclusion that it is not possible for him to be wrong. I personally
>> attribute this to the fact that he's most likely an advanced robot
>> from the future. That said, Noah has voiced concerns to various ideas
>> and we should make sure that any of his concerns are fully addressed.
>>
>> We should attempt to make sure that any tool support doesn't morph
>> into tool requirement. For instance, I think we should make sure that
>> its possible to keep compiling CouchDB without rebar and not come to
>> rely on it.
>>
>> While I'd be more than happy to start in on this and handle all of the
>> build system refactoring to make this happen, I'm not going to start
>> until there's a community consensus on what needs to be done. There
>> are a couple paths that I could see us taking to make this happen. We
>> could just make the current source tree be rebar compatible and figure
>> out the build system to do the optional rebar build or we could also
>> take this chance to split the source code into multiple applications.
>> Personally, I'd prefer to take this opportunity to organize the code
>> with multiple erlang apps.
>>
>> Too get the conversation rolling here's a first pass at a new app proposal:
>>
>> etap:
>>
>> Nick Gerakines now releases etap as a single .erl file that can be
>> dropped into the test directory. This app should be removed in favor
>> of that method.
>>
>> erlang-oauth:
>>
>> Should be renamed to just oauth. That erlang- prefix has bugged me
>> fore entirely too long.
>>
>> mochiweb, ibrowse, oauth:
>>
>> Refactored to use standard src, include, ebin, priv directories to
>> be OTP compliant. This results in directories like
>>
>> ./src/$APP/ebin
>> ./src/$APP/incldue
>> ./src/$APP/priv
>> ./src/$APP/src
>>
>> couchdb:
>>
>> Each proposed app will be structured as described above. Proposed apps:
>>
>> couch_core: The core Erlang modules for storing docs and managing
>> "internal infrastructure"
>> couch_view: The view engine as well as the holder for managing OS processes.
>> couch_rep: couch_rep*.erl
>> couch_externals: couch_external*.erl
>> couch_httpd: couch_http*.erl
>>
>> At the bottom of this email I made an initial pass through the
>> ./src/couchdb tree to classify file by file into the described apps.
>> There are also some minor warts in this split. Things like the core
>> couchdb classes currently require calling into the couch_view app and
>> what not. Given the "interesting" call graph for CouchDb internals, I
>> think we should avoid addressing this and aim to minimize the actual
>> change to source file contents during this move. After modules have
>> been moved we can then start on teasing out the various cyclic
>> dependencies as necessary.
>>
>> Also, on a last note, seeing as this is a major change to the source
>> tree (and ideally 0 change to source code) it might be a good idea to
>> start this work just after we release 1.1.0.
>>
>>
>> What do you guys think?
>>
>> Paul Davis
>>
>>
>> List of files by per app:
>>
>> couch_core:
>>
>> ./src/couchdb/couch.erl
>> ./src/couchdb/couch_app.erl
>> ./src/couchdb/couch_auth_cache.erl
>> ./src/couchdb/couch_btree.erl
>> ./src/couchdb/couch_changes.erl
>> ./src/couchdb/couch_config.erl
>> ./src/couchdb/couch_config_writer.erl
>> ./src/couchdb/couch_db.erl
>> ./src/couchdb/couch_db.hrl
>> ./src/couchdb/couch_db_update_notifier.erl
>> ./src/couchdb/couch_db_update_notifier_sup.erl
>> ./src/couchdb/couch_db_updater.erl
>> ./src/couchdb/couch_doc.erl
>> ./src/couchdb/couch_event_sup.erl
>> ./src/couchdb/couch_file.erl
>> ./src/couchdb/couch_key_tree.erl
>> ./src/couchdb/couch_log.erl
>> ./src/couchdb/couch_ref_counter.erl
>> ./src/couchdb/couch_server.erl
>> ./src/couchdb/couch_server_sup.erl
>> ./src/couchdb/couch_stats_aggregator.erl
>> ./src/couchdb/couch_stats_collector.erl
>> ./src/couchdb/couch_stream.erl
>> ./src/couchdb/couch_task_status.erl
>> ./src/couchdb/couch_util.erl
>> ./src/couchdb/couch_uuids.erl
>> ./src/couchdb/priv/icu_driver/couch_icu_driver.c
>> ./src/couchdb/priv/stat_descriptions.cfg.in
>>
>> couch_view:
>>
>> ./src/couchdb/couch_native_process.erl
>> ./src/couchdb/couch_os_process.erl
>> ./src/couchdb/couch_query_servers.erl
>> ./src/couchdb/couch_view.erl
>> ./src/couchdb/couch_view_compactor.erl
>> ./src/couchdb/couch_view_group.erl
>> ./src/couchdb/couch_view_updater.erl
>> ./src/couchdb/couch_work_queue.erl
>> ./src/couchdb/priv/couch_js/http.c
>> ./src/couchdb/priv/couch_js/http.h
>> ./src/couchdb/priv/couch_js/main.c
>> ./src/couchdb/priv/couch_js/utf8.c
>> ./src/couchdb/priv/couch_js/utf8.h
>> ./src/couchdb/priv/spawnkillable/couchspawnkillable.sh
>> ./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
>>
>> couch_rep:
>>
>> ./src/couchdb/couch_js_functions.hrl
>> ./src/couchdb/couch_rep.erl
>> ./src/couchdb/couch_rep_att.erl
>> ./src/couchdb/couch_rep_changes_feed.erl
>> ./src/couchdb/couch_rep_db_listener.erl
>> ./src/couchdb/couch_rep_httpc.erl
>> ./src/couchdb/couch_rep_missing_revs.erl
>> ./src/couchdb/couch_rep_reader.erl
>> ./src/couchdb/couch_rep_sup.erl
>> ./src/couchdb/couch_rep_writer.erl
>>
>> couch_externals:
>>
>> ./src/couchdb/couch_external_manager.erl
>> ./src/couchdb/couch_external_server.erl
>>
>> couch_httpd:
>>
>> ./src/couchdb/couch_httpd.erl
>> ./src/couchdb/couch_httpd_auth.erl
>> ./src/couchdb/couch_httpd_db.erl
>> ./src/couchdb/couch_httpd_external.erl
>> ./src/couchdb/couch_httpd_misc_handlers.erl
>> ./src/couchdb/couch_httpd_oauth.erl
>> ./src/couchdb/couch_httpd_rewrite.erl
>> ./src/couchdb/couch_httpd_show.erl
>> ./src/couchdb/couch_httpd_stats_handlers.erl
>> ./src/couchdb/couch_httpd_vhost.erl
>> ./src/couchdb/couch_httpd_view.erl
>>
Re: Using rebar to install couchdb
Posted by Robert Newson <ro...@gmail.com>.
Paul,
Brilliant writeup and proposal. I'd like to see all those things
happen pretty much as you said. Cleaning the cycles out will be much
easier once things are broken out in that style.
+1
B.
On Thu, Oct 14, 2010 at 7:54 PM, Paul Davis <pa...@gmail.com> wrote:
> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>> In an attempt to start some merging with cloudant I would like to
>> start by using rebar in our install process.
>>
>> Like i see it, we could continue to use autotools to create the
>> rebar.config files and other templates an then rebar for the final
>> build and dependencies management. This changes as noticed by @davisp
>> also imply we make our tree a little more OTP compliant. I would like
>> to start this work asap.
>>
>> Thoughts ?
>>
>> - benoit
>>
>
> So there's a couple issues at hand here which seem to be motivated by
> the desire to start using tools like rebar.
>
> Our current source tree is not compliant with some of the basic
> Erlang/OTP conventions. This is both bad technically and socially.
> Technically, it prevents us from easily integrating tools like rebar
> that would help advanced users with things like making Erlang reltools
> packages. Socially, it doesn't reflect well on us to members of the
> Erlang community that may have otherwise become contributors. All
> languages have a standard package layout and Erlang is no different.
>
> The current CouchDB Erlang app has grown considerably. There's been
> general consensus that we need to start splitting it up into smaller
> applications that encompass specific functionality. There's been a bit
> of effort in this direction, but its such a major change to source
> file location it needs to have a community consensus to really start
> working on seriously.
>
> I don't think we should focus directly on the issue of integrating
> rebar. It should definitely be a goal, but not at the cost of our
> current situation. Noah Slater has maintained an excellent build
> system for us as is shown by the number of people building CouchDB
> from source and the number of packages available. While I have argued
> with him on numerous occasions about details, I have come to the
> conclusion that it is not possible for him to be wrong. I personally
> attribute this to the fact that he's most likely an advanced robot
> from the future. That said, Noah has voiced concerns to various ideas
> and we should make sure that any of his concerns are fully addressed.
>
> We should attempt to make sure that any tool support doesn't morph
> into tool requirement. For instance, I think we should make sure that
> its possible to keep compiling CouchDB without rebar and not come to
> rely on it.
>
> While I'd be more than happy to start in on this and handle all of the
> build system refactoring to make this happen, I'm not going to start
> until there's a community consensus on what needs to be done. There
> are a couple paths that I could see us taking to make this happen. We
> could just make the current source tree be rebar compatible and figure
> out the build system to do the optional rebar build or we could also
> take this chance to split the source code into multiple applications.
> Personally, I'd prefer to take this opportunity to organize the code
> with multiple erlang apps.
>
> Too get the conversation rolling here's a first pass at a new app proposal:
>
> etap:
>
> Nick Gerakines now releases etap as a single .erl file that can be
> dropped into the test directory. This app should be removed in favor
> of that method.
>
> erlang-oauth:
>
> Should be renamed to just oauth. That erlang- prefix has bugged me
> fore entirely too long.
>
> mochiweb, ibrowse, oauth:
>
> Refactored to use standard src, include, ebin, priv directories to
> be OTP compliant. This results in directories like
>
> ./src/$APP/ebin
> ./src/$APP/incldue
> ./src/$APP/priv
> ./src/$APP/src
>
> couchdb:
>
> Each proposed app will be structured as described above. Proposed apps:
>
> couch_core: The core Erlang modules for storing docs and managing
> "internal infrastructure"
> couch_view: The view engine as well as the holder for managing OS processes.
> couch_rep: couch_rep*.erl
> couch_externals: couch_external*.erl
> couch_httpd: couch_http*.erl
>
> At the bottom of this email I made an initial pass through the
> ./src/couchdb tree to classify file by file into the described apps.
> There are also some minor warts in this split. Things like the core
> couchdb classes currently require calling into the couch_view app and
> what not. Given the "interesting" call graph for CouchDb internals, I
> think we should avoid addressing this and aim to minimize the actual
> change to source file contents during this move. After modules have
> been moved we can then start on teasing out the various cyclic
> dependencies as necessary.
>
> Also, on a last note, seeing as this is a major change to the source
> tree (and ideally 0 change to source code) it might be a good idea to
> start this work just after we release 1.1.0.
>
>
> What do you guys think?
>
> Paul Davis
>
>
> List of files by per app:
>
> couch_core:
>
> ./src/couchdb/couch.erl
> ./src/couchdb/couch_app.erl
> ./src/couchdb/couch_auth_cache.erl
> ./src/couchdb/couch_btree.erl
> ./src/couchdb/couch_changes.erl
> ./src/couchdb/couch_config.erl
> ./src/couchdb/couch_config_writer.erl
> ./src/couchdb/couch_db.erl
> ./src/couchdb/couch_db.hrl
> ./src/couchdb/couch_db_update_notifier.erl
> ./src/couchdb/couch_db_update_notifier_sup.erl
> ./src/couchdb/couch_db_updater.erl
> ./src/couchdb/couch_doc.erl
> ./src/couchdb/couch_event_sup.erl
> ./src/couchdb/couch_file.erl
> ./src/couchdb/couch_key_tree.erl
> ./src/couchdb/couch_log.erl
> ./src/couchdb/couch_ref_counter.erl
> ./src/couchdb/couch_server.erl
> ./src/couchdb/couch_server_sup.erl
> ./src/couchdb/couch_stats_aggregator.erl
> ./src/couchdb/couch_stats_collector.erl
> ./src/couchdb/couch_stream.erl
> ./src/couchdb/couch_task_status.erl
> ./src/couchdb/couch_util.erl
> ./src/couchdb/couch_uuids.erl
> ./src/couchdb/priv/icu_driver/couch_icu_driver.c
> ./src/couchdb/priv/stat_descriptions.cfg.in
>
> couch_view:
>
> ./src/couchdb/couch_native_process.erl
> ./src/couchdb/couch_os_process.erl
> ./src/couchdb/couch_query_servers.erl
> ./src/couchdb/couch_view.erl
> ./src/couchdb/couch_view_compactor.erl
> ./src/couchdb/couch_view_group.erl
> ./src/couchdb/couch_view_updater.erl
> ./src/couchdb/couch_work_queue.erl
> ./src/couchdb/priv/couch_js/http.c
> ./src/couchdb/priv/couch_js/http.h
> ./src/couchdb/priv/couch_js/main.c
> ./src/couchdb/priv/couch_js/utf8.c
> ./src/couchdb/priv/couch_js/utf8.h
> ./src/couchdb/priv/spawnkillable/couchspawnkillable.sh
> ./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
>
> couch_rep:
>
> ./src/couchdb/couch_js_functions.hrl
> ./src/couchdb/couch_rep.erl
> ./src/couchdb/couch_rep_att.erl
> ./src/couchdb/couch_rep_changes_feed.erl
> ./src/couchdb/couch_rep_db_listener.erl
> ./src/couchdb/couch_rep_httpc.erl
> ./src/couchdb/couch_rep_missing_revs.erl
> ./src/couchdb/couch_rep_reader.erl
> ./src/couchdb/couch_rep_sup.erl
> ./src/couchdb/couch_rep_writer.erl
>
> couch_externals:
>
> ./src/couchdb/couch_external_manager.erl
> ./src/couchdb/couch_external_server.erl
>
> couch_httpd:
>
> ./src/couchdb/couch_httpd.erl
> ./src/couchdb/couch_httpd_auth.erl
> ./src/couchdb/couch_httpd_db.erl
> ./src/couchdb/couch_httpd_external.erl
> ./src/couchdb/couch_httpd_misc_handlers.erl
> ./src/couchdb/couch_httpd_oauth.erl
> ./src/couchdb/couch_httpd_rewrite.erl
> ./src/couchdb/couch_httpd_show.erl
> ./src/couchdb/couch_httpd_stats_handlers.erl
> ./src/couchdb/couch_httpd_vhost.erl
> ./src/couchdb/couch_httpd_view.erl
>
Re: Using rebar to install couchdb
Posted by Jan Lehnardt <ja...@apache.org>.
On 18 Oct 2010, at 03:03, Paul Davis wrote:
> On Sun, Oct 17, 2010 at 3:20 PM, Jan Lehnardt <ja...@apache.org> wrote:
>> Ace, Paul :)
>>
>> Just some bikeshedding as I agree with most that is said here.
>>
>> I'm wondering if the OS process handling could be its own module or
>> part of couch_core. While the View Server is the primary user, it is
>> not the only one and having OS process management in the view server
>> module just feels wrong to me :)
>>
>> Cheers
>> Jan
>> --
>>
>
> Damn. I was kinda hoping no one would notice that until after I had an
> answer. Thanks for ruining my plan.
Sorry :)
I think your plan of doing the separation separately from rearchitecing
the source is solid.
Cheers
Jan
--
>
> I've been thinking over it for the last few days and my current
> favorite idea is mostly similar to what you suggest in making an app
> that does generic OS process handling like we're discussing in the
> thread about refactoring couch_query_servers.erl. The issue at hand is
> figuring out a method to make this mix in well with our current
> situation of reusing those OS processes for multiple duties. My
> general thinking in this direction is that this API will end up
> evolving a bit to handle the ability to be extended as we add new OS
> level functionality.
>
> My primary goal for the source tree layout is to change as little
> source code as possible so I was planning on delaying internal
> refactoring until after we land the retooling of the build mechanics
> so that our svn history retains some semblance of sanity as we move
> through this.
>
> My current plan of attack is to hack a shell script and a couple small
> diffs to make the source move. Once I get that project up and running
> I'll start a JIRA ticket so we have a place for focusing detail
> oriented discussions.
>
> Paul Davis
>
>
>>
>>
>> On 14 Oct 2010, at 20:54, Paul Davis wrote:
>>
>>> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>>>> In an attempt to start some merging with cloudant I would like to
>>>> start by using rebar in our install process.
>>>>
>>>> Like i see it, we could continue to use autotools to create the
>>>> rebar.config files and other templates an then rebar for the final
>>>> build and dependencies management. This changes as noticed by @davisp
>>>> also imply we make our tree a little more OTP compliant. I would like
>>>> to start this work asap.
>>>>
>>>> Thoughts ?
>>>>
>>>> - benoit
>>>>
>>>
>>> So there's a couple issues at hand here which seem to be motivated by
>>> the desire to start using tools like rebar.
>>>
>>> Our current source tree is not compliant with some of the basic
>>> Erlang/OTP conventions. This is both bad technically and socially.
>>> Technically, it prevents us from easily integrating tools like rebar
>>> that would help advanced users with things like making Erlang reltools
>>> packages. Socially, it doesn't reflect well on us to members of the
>>> Erlang community that may have otherwise become contributors. All
>>> languages have a standard package layout and Erlang is no different.
>>>
>>> The current CouchDB Erlang app has grown considerably. There's been
>>> general consensus that we need to start splitting it up into smaller
>>> applications that encompass specific functionality. There's been a bit
>>> of effort in this direction, but its such a major change to source
>>> file location it needs to have a community consensus to really start
>>> working on seriously.
>>>
>>> I don't think we should focus directly on the issue of integrating
>>> rebar. It should definitely be a goal, but not at the cost of our
>>> current situation. Noah Slater has maintained an excellent build
>>> system for us as is shown by the number of people building CouchDB
>>> from source and the number of packages available. While I have argued
>>> with him on numerous occasions about details, I have come to the
>>> conclusion that it is not possible for him to be wrong. I personally
>>> attribute this to the fact that he's most likely an advanced robot
>>> from the future. That said, Noah has voiced concerns to various ideas
>>> and we should make sure that any of his concerns are fully addressed.
>>>
>>> We should attempt to make sure that any tool support doesn't morph
>>> into tool requirement. For instance, I think we should make sure that
>>> its possible to keep compiling CouchDB without rebar and not come to
>>> rely on it.
>>>
>>> While I'd be more than happy to start in on this and handle all of the
>>> build system refactoring to make this happen, I'm not going to start
>>> until there's a community consensus on what needs to be done. There
>>> are a couple paths that I could see us taking to make this happen. We
>>> could just make the current source tree be rebar compatible and figure
>>> out the build system to do the optional rebar build or we could also
>>> take this chance to split the source code into multiple applications.
>>> Personally, I'd prefer to take this opportunity to organize the code
>>> with multiple erlang apps.
>>>
>>> Too get the conversation rolling here's a first pass at a new app proposal:
>>>
>>> etap:
>>>
>>> Nick Gerakines now releases etap as a single .erl file that can be
>>> dropped into the test directory. This app should be removed in favor
>>> of that method.
>>>
>>> erlang-oauth:
>>>
>>> Should be renamed to just oauth. That erlang- prefix has bugged me
>>> fore entirely too long.
>>>
>>> mochiweb, ibrowse, oauth:
>>>
>>> Refactored to use standard src, include, ebin, priv directories to
>>> be OTP compliant. This results in directories like
>>>
>>> ./src/$APP/ebin
>>> ./src/$APP/incldue
>>> ./src/$APP/priv
>>> ./src/$APP/src
>>>
>>> couchdb:
>>>
>>> Each proposed app will be structured as described above. Proposed apps:
>>>
>>> couch_core: The core Erlang modules for storing docs and managing
>>> "internal infrastructure"
>>> couch_view: The view engine as well as the holder for managing OS processes.
>>> couch_rep: couch_rep*.erl
>>> couch_externals: couch_external*.erl
>>> couch_httpd: couch_http*.erl
>>>
>>> At the bottom of this email I made an initial pass through the
>>> ./src/couchdb tree to classify file by file into the described apps.
>>> There are also some minor warts in this split. Things like the core
>>> couchdb classes currently require calling into the couch_view app and
>>> what not. Given the "interesting" call graph for CouchDb internals, I
>>> think we should avoid addressing this and aim to minimize the actual
>>> change to source file contents during this move. After modules have
>>> been moved we can then start on teasing out the various cyclic
>>> dependencies as necessary.
>>>
>>> Also, on a last note, seeing as this is a major change to the source
>>> tree (and ideally 0 change to source code) it might be a good idea to
>>> start this work just after we release 1.1.0.
>>>
>>>
>>> What do you guys think?
>>>
>>> Paul Davis
>>>
>>>
>>> List of files by per app:
>>>
>>> couch_core:
>>>
>>> ./src/couchdb/couch.erl
>>> ./src/couchdb/couch_app.erl
>>> ./src/couchdb/couch_auth_cache.erl
>>> ./src/couchdb/couch_btree.erl
>>> ./src/couchdb/couch_changes.erl
>>> ./src/couchdb/couch_config.erl
>>> ./src/couchdb/couch_config_writer.erl
>>> ./src/couchdb/couch_db.erl
>>> ./src/couchdb/couch_db.hrl
>>> ./src/couchdb/couch_db_update_notifier.erl
>>> ./src/couchdb/couch_db_update_notifier_sup.erl
>>> ./src/couchdb/couch_db_updater.erl
>>> ./src/couchdb/couch_doc.erl
>>> ./src/couchdb/couch_event_sup.erl
>>> ./src/couchdb/couch_file.erl
>>> ./src/couchdb/couch_key_tree.erl
>>> ./src/couchdb/couch_log.erl
>>> ./src/couchdb/couch_ref_counter.erl
>>> ./src/couchdb/couch_server.erl
>>> ./src/couchdb/couch_server_sup.erl
>>> ./src/couchdb/couch_stats_aggregator.erl
>>> ./src/couchdb/couch_stats_collector.erl
>>> ./src/couchdb/couch_stream.erl
>>> ./src/couchdb/couch_task_status.erl
>>> ./src/couchdb/couch_util.erl
>>> ./src/couchdb/couch_uuids.erl
>>> ./src/couchdb/priv/icu_driver/couch_icu_driver.c
>>> ./src/couchdb/priv/stat_descriptions.cfg.in
>>>
>>> couch_view:
>>>
>>> ./src/couchdb/couch_native_process.erl
>>> ./src/couchdb/couch_os_process.erl
>>> ./src/couchdb/couch_query_servers.erl
>>> ./src/couchdb/couch_view.erl
>>> ./src/couchdb/couch_view_compactor.erl
>>> ./src/couchdb/couch_view_group.erl
>>> ./src/couchdb/couch_view_updater.erl
>>> ./src/couchdb/couch_work_queue.erl
>>> ./src/couchdb/priv/couch_js/http.c
>>> ./src/couchdb/priv/couch_js/http.h
>>> ./src/couchdb/priv/couch_js/main.c
>>> ./src/couchdb/priv/couch_js/utf8.c
>>> ./src/couchdb/priv/couch_js/utf8.h
>>> ./src/couchdb/priv/spawnkillable/couchspawnkillable.sh
>>> ./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
>>>
>>> couch_rep:
>>>
>>> ./src/couchdb/couch_js_functions.hrl
>>> ./src/couchdb/couch_rep.erl
>>> ./src/couchdb/couch_rep_att.erl
>>> ./src/couchdb/couch_rep_changes_feed.erl
>>> ./src/couchdb/couch_rep_db_listener.erl
>>> ./src/couchdb/couch_rep_httpc.erl
>>> ./src/couchdb/couch_rep_missing_revs.erl
>>> ./src/couchdb/couch_rep_reader.erl
>>> ./src/couchdb/couch_rep_sup.erl
>>> ./src/couchdb/couch_rep_writer.erl
>>>
>>> couch_externals:
>>>
>>> ./src/couchdb/couch_external_manager.erl
>>> ./src/couchdb/couch_external_server.erl
>>>
>>> couch_httpd:
>>>
>>> ./src/couchdb/couch_httpd.erl
>>> ./src/couchdb/couch_httpd_auth.erl
>>> ./src/couchdb/couch_httpd_db.erl
>>> ./src/couchdb/couch_httpd_external.erl
>>> ./src/couchdb/couch_httpd_misc_handlers.erl
>>> ./src/couchdb/couch_httpd_oauth.erl
>>> ./src/couchdb/couch_httpd_rewrite.erl
>>> ./src/couchdb/couch_httpd_show.erl
>>> ./src/couchdb/couch_httpd_stats_handlers.erl
>>> ./src/couchdb/couch_httpd_vhost.erl
>>> ./src/couchdb/couch_httpd_view.erl
>>
>>
Re: Using rebar to install couchdb
Posted by Paul Davis <pa...@gmail.com>.
On Sun, Oct 17, 2010 at 3:20 PM, Jan Lehnardt <ja...@apache.org> wrote:
> Ace, Paul :)
>
> Just some bikeshedding as I agree with most that is said here.
>
> I'm wondering if the OS process handling could be its own module or
> part of couch_core. While the View Server is the primary user, it is
> not the only one and having OS process management in the view server
> module just feels wrong to me :)
>
> Cheers
> Jan
> --
>
Damn. I was kinda hoping no one would notice that until after I had an
answer. Thanks for ruining my plan.
I've been thinking over it for the last few days and my current
favorite idea is mostly similar to what you suggest in making an app
that does generic OS process handling like we're discussing in the
thread about refactoring couch_query_servers.erl. The issue at hand is
figuring out a method to make this mix in well with our current
situation of reusing those OS processes for multiple duties. My
general thinking in this direction is that this API will end up
evolving a bit to handle the ability to be extended as we add new OS
level functionality.
My primary goal for the source tree layout is to change as little
source code as possible so I was planning on delaying internal
refactoring until after we land the retooling of the build mechanics
so that our svn history retains some semblance of sanity as we move
through this.
My current plan of attack is to hack a shell script and a couple small
diffs to make the source move. Once I get that project up and running
I'll start a JIRA ticket so we have a place for focusing detail
oriented discussions.
Paul Davis
>
>
> On 14 Oct 2010, at 20:54, Paul Davis wrote:
>
>> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>>> In an attempt to start some merging with cloudant I would like to
>>> start by using rebar in our install process.
>>>
>>> Like i see it, we could continue to use autotools to create the
>>> rebar.config files and other templates an then rebar for the final
>>> build and dependencies management. This changes as noticed by @davisp
>>> also imply we make our tree a little more OTP compliant. I would like
>>> to start this work asap.
>>>
>>> Thoughts ?
>>>
>>> - benoit
>>>
>>
>> So there's a couple issues at hand here which seem to be motivated by
>> the desire to start using tools like rebar.
>>
>> Our current source tree is not compliant with some of the basic
>> Erlang/OTP conventions. This is both bad technically and socially.
>> Technically, it prevents us from easily integrating tools like rebar
>> that would help advanced users with things like making Erlang reltools
>> packages. Socially, it doesn't reflect well on us to members of the
>> Erlang community that may have otherwise become contributors. All
>> languages have a standard package layout and Erlang is no different.
>>
>> The current CouchDB Erlang app has grown considerably. There's been
>> general consensus that we need to start splitting it up into smaller
>> applications that encompass specific functionality. There's been a bit
>> of effort in this direction, but its such a major change to source
>> file location it needs to have a community consensus to really start
>> working on seriously.
>>
>> I don't think we should focus directly on the issue of integrating
>> rebar. It should definitely be a goal, but not at the cost of our
>> current situation. Noah Slater has maintained an excellent build
>> system for us as is shown by the number of people building CouchDB
>> from source and the number of packages available. While I have argued
>> with him on numerous occasions about details, I have come to the
>> conclusion that it is not possible for him to be wrong. I personally
>> attribute this to the fact that he's most likely an advanced robot
>> from the future. That said, Noah has voiced concerns to various ideas
>> and we should make sure that any of his concerns are fully addressed.
>>
>> We should attempt to make sure that any tool support doesn't morph
>> into tool requirement. For instance, I think we should make sure that
>> its possible to keep compiling CouchDB without rebar and not come to
>> rely on it.
>>
>> While I'd be more than happy to start in on this and handle all of the
>> build system refactoring to make this happen, I'm not going to start
>> until there's a community consensus on what needs to be done. There
>> are a couple paths that I could see us taking to make this happen. We
>> could just make the current source tree be rebar compatible and figure
>> out the build system to do the optional rebar build or we could also
>> take this chance to split the source code into multiple applications.
>> Personally, I'd prefer to take this opportunity to organize the code
>> with multiple erlang apps.
>>
>> Too get the conversation rolling here's a first pass at a new app proposal:
>>
>> etap:
>>
>> Nick Gerakines now releases etap as a single .erl file that can be
>> dropped into the test directory. This app should be removed in favor
>> of that method.
>>
>> erlang-oauth:
>>
>> Should be renamed to just oauth. That erlang- prefix has bugged me
>> fore entirely too long.
>>
>> mochiweb, ibrowse, oauth:
>>
>> Refactored to use standard src, include, ebin, priv directories to
>> be OTP compliant. This results in directories like
>>
>> ./src/$APP/ebin
>> ./src/$APP/incldue
>> ./src/$APP/priv
>> ./src/$APP/src
>>
>> couchdb:
>>
>> Each proposed app will be structured as described above. Proposed apps:
>>
>> couch_core: The core Erlang modules for storing docs and managing
>> "internal infrastructure"
>> couch_view: The view engine as well as the holder for managing OS processes.
>> couch_rep: couch_rep*.erl
>> couch_externals: couch_external*.erl
>> couch_httpd: couch_http*.erl
>>
>> At the bottom of this email I made an initial pass through the
>> ./src/couchdb tree to classify file by file into the described apps.
>> There are also some minor warts in this split. Things like the core
>> couchdb classes currently require calling into the couch_view app and
>> what not. Given the "interesting" call graph for CouchDb internals, I
>> think we should avoid addressing this and aim to minimize the actual
>> change to source file contents during this move. After modules have
>> been moved we can then start on teasing out the various cyclic
>> dependencies as necessary.
>>
>> Also, on a last note, seeing as this is a major change to the source
>> tree (and ideally 0 change to source code) it might be a good idea to
>> start this work just after we release 1.1.0.
>>
>>
>> What do you guys think?
>>
>> Paul Davis
>>
>>
>> List of files by per app:
>>
>> couch_core:
>>
>> ./src/couchdb/couch.erl
>> ./src/couchdb/couch_app.erl
>> ./src/couchdb/couch_auth_cache.erl
>> ./src/couchdb/couch_btree.erl
>> ./src/couchdb/couch_changes.erl
>> ./src/couchdb/couch_config.erl
>> ./src/couchdb/couch_config_writer.erl
>> ./src/couchdb/couch_db.erl
>> ./src/couchdb/couch_db.hrl
>> ./src/couchdb/couch_db_update_notifier.erl
>> ./src/couchdb/couch_db_update_notifier_sup.erl
>> ./src/couchdb/couch_db_updater.erl
>> ./src/couchdb/couch_doc.erl
>> ./src/couchdb/couch_event_sup.erl
>> ./src/couchdb/couch_file.erl
>> ./src/couchdb/couch_key_tree.erl
>> ./src/couchdb/couch_log.erl
>> ./src/couchdb/couch_ref_counter.erl
>> ./src/couchdb/couch_server.erl
>> ./src/couchdb/couch_server_sup.erl
>> ./src/couchdb/couch_stats_aggregator.erl
>> ./src/couchdb/couch_stats_collector.erl
>> ./src/couchdb/couch_stream.erl
>> ./src/couchdb/couch_task_status.erl
>> ./src/couchdb/couch_util.erl
>> ./src/couchdb/couch_uuids.erl
>> ./src/couchdb/priv/icu_driver/couch_icu_driver.c
>> ./src/couchdb/priv/stat_descriptions.cfg.in
>>
>> couch_view:
>>
>> ./src/couchdb/couch_native_process.erl
>> ./src/couchdb/couch_os_process.erl
>> ./src/couchdb/couch_query_servers.erl
>> ./src/couchdb/couch_view.erl
>> ./src/couchdb/couch_view_compactor.erl
>> ./src/couchdb/couch_view_group.erl
>> ./src/couchdb/couch_view_updater.erl
>> ./src/couchdb/couch_work_queue.erl
>> ./src/couchdb/priv/couch_js/http.c
>> ./src/couchdb/priv/couch_js/http.h
>> ./src/couchdb/priv/couch_js/main.c
>> ./src/couchdb/priv/couch_js/utf8.c
>> ./src/couchdb/priv/couch_js/utf8.h
>> ./src/couchdb/priv/spawnkillable/couchspawnkillable.sh
>> ./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
>>
>> couch_rep:
>>
>> ./src/couchdb/couch_js_functions.hrl
>> ./src/couchdb/couch_rep.erl
>> ./src/couchdb/couch_rep_att.erl
>> ./src/couchdb/couch_rep_changes_feed.erl
>> ./src/couchdb/couch_rep_db_listener.erl
>> ./src/couchdb/couch_rep_httpc.erl
>> ./src/couchdb/couch_rep_missing_revs.erl
>> ./src/couchdb/couch_rep_reader.erl
>> ./src/couchdb/couch_rep_sup.erl
>> ./src/couchdb/couch_rep_writer.erl
>>
>> couch_externals:
>>
>> ./src/couchdb/couch_external_manager.erl
>> ./src/couchdb/couch_external_server.erl
>>
>> couch_httpd:
>>
>> ./src/couchdb/couch_httpd.erl
>> ./src/couchdb/couch_httpd_auth.erl
>> ./src/couchdb/couch_httpd_db.erl
>> ./src/couchdb/couch_httpd_external.erl
>> ./src/couchdb/couch_httpd_misc_handlers.erl
>> ./src/couchdb/couch_httpd_oauth.erl
>> ./src/couchdb/couch_httpd_rewrite.erl
>> ./src/couchdb/couch_httpd_show.erl
>> ./src/couchdb/couch_httpd_stats_handlers.erl
>> ./src/couchdb/couch_httpd_vhost.erl
>> ./src/couchdb/couch_httpd_view.erl
>
>
Re: Using rebar to install couchdb
Posted by Jan Lehnardt <ja...@apache.org>.
Ace, Paul :)
Just some bikeshedding as I agree with most that is said here.
I'm wondering if the OS process handling could be its own module or
part of couch_core. While the View Server is the primary user, it is
not the only one and having OS process management in the view server
module just feels wrong to me :)
Cheers
Jan
--
On 14 Oct 2010, at 20:54, Paul Davis wrote:
> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>> In an attempt to start some merging with cloudant I would like to
>> start by using rebar in our install process.
>>
>> Like i see it, we could continue to use autotools to create the
>> rebar.config files and other templates an then rebar for the final
>> build and dependencies management. This changes as noticed by @davisp
>> also imply we make our tree a little more OTP compliant. I would like
>> to start this work asap.
>>
>> Thoughts ?
>>
>> - benoit
>>
>
> So there's a couple issues at hand here which seem to be motivated by
> the desire to start using tools like rebar.
>
> Our current source tree is not compliant with some of the basic
> Erlang/OTP conventions. This is both bad technically and socially.
> Technically, it prevents us from easily integrating tools like rebar
> that would help advanced users with things like making Erlang reltools
> packages. Socially, it doesn't reflect well on us to members of the
> Erlang community that may have otherwise become contributors. All
> languages have a standard package layout and Erlang is no different.
>
> The current CouchDB Erlang app has grown considerably. There's been
> general consensus that we need to start splitting it up into smaller
> applications that encompass specific functionality. There's been a bit
> of effort in this direction, but its such a major change to source
> file location it needs to have a community consensus to really start
> working on seriously.
>
> I don't think we should focus directly on the issue of integrating
> rebar. It should definitely be a goal, but not at the cost of our
> current situation. Noah Slater has maintained an excellent build
> system for us as is shown by the number of people building CouchDB
> from source and the number of packages available. While I have argued
> with him on numerous occasions about details, I have come to the
> conclusion that it is not possible for him to be wrong. I personally
> attribute this to the fact that he's most likely an advanced robot
> from the future. That said, Noah has voiced concerns to various ideas
> and we should make sure that any of his concerns are fully addressed.
>
> We should attempt to make sure that any tool support doesn't morph
> into tool requirement. For instance, I think we should make sure that
> its possible to keep compiling CouchDB without rebar and not come to
> rely on it.
>
> While I'd be more than happy to start in on this and handle all of the
> build system refactoring to make this happen, I'm not going to start
> until there's a community consensus on what needs to be done. There
> are a couple paths that I could see us taking to make this happen. We
> could just make the current source tree be rebar compatible and figure
> out the build system to do the optional rebar build or we could also
> take this chance to split the source code into multiple applications.
> Personally, I'd prefer to take this opportunity to organize the code
> with multiple erlang apps.
>
> Too get the conversation rolling here's a first pass at a new app proposal:
>
> etap:
>
> Nick Gerakines now releases etap as a single .erl file that can be
> dropped into the test directory. This app should be removed in favor
> of that method.
>
> erlang-oauth:
>
> Should be renamed to just oauth. That erlang- prefix has bugged me
> fore entirely too long.
>
> mochiweb, ibrowse, oauth:
>
> Refactored to use standard src, include, ebin, priv directories to
> be OTP compliant. This results in directories like
>
> ./src/$APP/ebin
> ./src/$APP/incldue
> ./src/$APP/priv
> ./src/$APP/src
>
> couchdb:
>
> Each proposed app will be structured as described above. Proposed apps:
>
> couch_core: The core Erlang modules for storing docs and managing
> "internal infrastructure"
> couch_view: The view engine as well as the holder for managing OS processes.
> couch_rep: couch_rep*.erl
> couch_externals: couch_external*.erl
> couch_httpd: couch_http*.erl
>
> At the bottom of this email I made an initial pass through the
> ./src/couchdb tree to classify file by file into the described apps.
> There are also some minor warts in this split. Things like the core
> couchdb classes currently require calling into the couch_view app and
> what not. Given the "interesting" call graph for CouchDb internals, I
> think we should avoid addressing this and aim to minimize the actual
> change to source file contents during this move. After modules have
> been moved we can then start on teasing out the various cyclic
> dependencies as necessary.
>
> Also, on a last note, seeing as this is a major change to the source
> tree (and ideally 0 change to source code) it might be a good idea to
> start this work just after we release 1.1.0.
>
>
> What do you guys think?
>
> Paul Davis
>
>
> List of files by per app:
>
> couch_core:
>
> ./src/couchdb/couch.erl
> ./src/couchdb/couch_app.erl
> ./src/couchdb/couch_auth_cache.erl
> ./src/couchdb/couch_btree.erl
> ./src/couchdb/couch_changes.erl
> ./src/couchdb/couch_config.erl
> ./src/couchdb/couch_config_writer.erl
> ./src/couchdb/couch_db.erl
> ./src/couchdb/couch_db.hrl
> ./src/couchdb/couch_db_update_notifier.erl
> ./src/couchdb/couch_db_update_notifier_sup.erl
> ./src/couchdb/couch_db_updater.erl
> ./src/couchdb/couch_doc.erl
> ./src/couchdb/couch_event_sup.erl
> ./src/couchdb/couch_file.erl
> ./src/couchdb/couch_key_tree.erl
> ./src/couchdb/couch_log.erl
> ./src/couchdb/couch_ref_counter.erl
> ./src/couchdb/couch_server.erl
> ./src/couchdb/couch_server_sup.erl
> ./src/couchdb/couch_stats_aggregator.erl
> ./src/couchdb/couch_stats_collector.erl
> ./src/couchdb/couch_stream.erl
> ./src/couchdb/couch_task_status.erl
> ./src/couchdb/couch_util.erl
> ./src/couchdb/couch_uuids.erl
> ./src/couchdb/priv/icu_driver/couch_icu_driver.c
> ./src/couchdb/priv/stat_descriptions.cfg.in
>
> couch_view:
>
> ./src/couchdb/couch_native_process.erl
> ./src/couchdb/couch_os_process.erl
> ./src/couchdb/couch_query_servers.erl
> ./src/couchdb/couch_view.erl
> ./src/couchdb/couch_view_compactor.erl
> ./src/couchdb/couch_view_group.erl
> ./src/couchdb/couch_view_updater.erl
> ./src/couchdb/couch_work_queue.erl
> ./src/couchdb/priv/couch_js/http.c
> ./src/couchdb/priv/couch_js/http.h
> ./src/couchdb/priv/couch_js/main.c
> ./src/couchdb/priv/couch_js/utf8.c
> ./src/couchdb/priv/couch_js/utf8.h
> ./src/couchdb/priv/spawnkillable/couchspawnkillable.sh
> ./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
>
> couch_rep:
>
> ./src/couchdb/couch_js_functions.hrl
> ./src/couchdb/couch_rep.erl
> ./src/couchdb/couch_rep_att.erl
> ./src/couchdb/couch_rep_changes_feed.erl
> ./src/couchdb/couch_rep_db_listener.erl
> ./src/couchdb/couch_rep_httpc.erl
> ./src/couchdb/couch_rep_missing_revs.erl
> ./src/couchdb/couch_rep_reader.erl
> ./src/couchdb/couch_rep_sup.erl
> ./src/couchdb/couch_rep_writer.erl
>
> couch_externals:
>
> ./src/couchdb/couch_external_manager.erl
> ./src/couchdb/couch_external_server.erl
>
> couch_httpd:
>
> ./src/couchdb/couch_httpd.erl
> ./src/couchdb/couch_httpd_auth.erl
> ./src/couchdb/couch_httpd_db.erl
> ./src/couchdb/couch_httpd_external.erl
> ./src/couchdb/couch_httpd_misc_handlers.erl
> ./src/couchdb/couch_httpd_oauth.erl
> ./src/couchdb/couch_httpd_rewrite.erl
> ./src/couchdb/couch_httpd_show.erl
> ./src/couchdb/couch_httpd_stats_handlers.erl
> ./src/couchdb/couch_httpd_vhost.erl
> ./src/couchdb/couch_httpd_view.erl
Re: Using rebar to install couchdb
Posted by Paul Davis <pa...@gmail.com>.
On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> In an attempt to start some merging with cloudant I would like to
> start by using rebar in our install process.
>
> Like i see it, we could continue to use autotools to create the
> rebar.config files and other templates an then rebar for the final
> build and dependencies management. This changes as noticed by @davisp
> also imply we make our tree a little more OTP compliant. I would like
> to start this work asap.
>
> Thoughts ?
>
> - benoit
>
So there's a couple issues at hand here which seem to be motivated by
the desire to start using tools like rebar.
Our current source tree is not compliant with some of the basic
Erlang/OTP conventions. This is both bad technically and socially.
Technically, it prevents us from easily integrating tools like rebar
that would help advanced users with things like making Erlang reltools
packages. Socially, it doesn't reflect well on us to members of the
Erlang community that may have otherwise become contributors. All
languages have a standard package layout and Erlang is no different.
The current CouchDB Erlang app has grown considerably. There's been
general consensus that we need to start splitting it up into smaller
applications that encompass specific functionality. There's been a bit
of effort in this direction, but its such a major change to source
file location it needs to have a community consensus to really start
working on seriously.
I don't think we should focus directly on the issue of integrating
rebar. It should definitely be a goal, but not at the cost of our
current situation. Noah Slater has maintained an excellent build
system for us as is shown by the number of people building CouchDB
from source and the number of packages available. While I have argued
with him on numerous occasions about details, I have come to the
conclusion that it is not possible for him to be wrong. I personally
attribute this to the fact that he's most likely an advanced robot
from the future. That said, Noah has voiced concerns to various ideas
and we should make sure that any of his concerns are fully addressed.
We should attempt to make sure that any tool support doesn't morph
into tool requirement. For instance, I think we should make sure that
its possible to keep compiling CouchDB without rebar and not come to
rely on it.
While I'd be more than happy to start in on this and handle all of the
build system refactoring to make this happen, I'm not going to start
until there's a community consensus on what needs to be done. There
are a couple paths that I could see us taking to make this happen. We
could just make the current source tree be rebar compatible and figure
out the build system to do the optional rebar build or we could also
take this chance to split the source code into multiple applications.
Personally, I'd prefer to take this opportunity to organize the code
with multiple erlang apps.
Too get the conversation rolling here's a first pass at a new app proposal:
etap:
Nick Gerakines now releases etap as a single .erl file that can be
dropped into the test directory. This app should be removed in favor
of that method.
erlang-oauth:
Should be renamed to just oauth. That erlang- prefix has bugged me
fore entirely too long.
mochiweb, ibrowse, oauth:
Refactored to use standard src, include, ebin, priv directories to
be OTP compliant. This results in directories like
./src/$APP/ebin
./src/$APP/incldue
./src/$APP/priv
./src/$APP/src
couchdb:
Each proposed app will be structured as described above. Proposed apps:
couch_core: The core Erlang modules for storing docs and managing
"internal infrastructure"
couch_view: The view engine as well as the holder for managing OS processes.
couch_rep: couch_rep*.erl
couch_externals: couch_external*.erl
couch_httpd: couch_http*.erl
At the bottom of this email I made an initial pass through the
./src/couchdb tree to classify file by file into the described apps.
There are also some minor warts in this split. Things like the core
couchdb classes currently require calling into the couch_view app and
what not. Given the "interesting" call graph for CouchDb internals, I
think we should avoid addressing this and aim to minimize the actual
change to source file contents during this move. After modules have
been moved we can then start on teasing out the various cyclic
dependencies as necessary.
Also, on a last note, seeing as this is a major change to the source
tree (and ideally 0 change to source code) it might be a good idea to
start this work just after we release 1.1.0.
What do you guys think?
Paul Davis
List of files by per app:
couch_core:
./src/couchdb/couch.erl
./src/couchdb/couch_app.erl
./src/couchdb/couch_auth_cache.erl
./src/couchdb/couch_btree.erl
./src/couchdb/couch_changes.erl
./src/couchdb/couch_config.erl
./src/couchdb/couch_config_writer.erl
./src/couchdb/couch_db.erl
./src/couchdb/couch_db.hrl
./src/couchdb/couch_db_update_notifier.erl
./src/couchdb/couch_db_update_notifier_sup.erl
./src/couchdb/couch_db_updater.erl
./src/couchdb/couch_doc.erl
./src/couchdb/couch_event_sup.erl
./src/couchdb/couch_file.erl
./src/couchdb/couch_key_tree.erl
./src/couchdb/couch_log.erl
./src/couchdb/couch_ref_counter.erl
./src/couchdb/couch_server.erl
./src/couchdb/couch_server_sup.erl
./src/couchdb/couch_stats_aggregator.erl
./src/couchdb/couch_stats_collector.erl
./src/couchdb/couch_stream.erl
./src/couchdb/couch_task_status.erl
./src/couchdb/couch_util.erl
./src/couchdb/couch_uuids.erl
./src/couchdb/priv/icu_driver/couch_icu_driver.c
./src/couchdb/priv/stat_descriptions.cfg.in
couch_view:
./src/couchdb/couch_native_process.erl
./src/couchdb/couch_os_process.erl
./src/couchdb/couch_query_servers.erl
./src/couchdb/couch_view.erl
./src/couchdb/couch_view_compactor.erl
./src/couchdb/couch_view_group.erl
./src/couchdb/couch_view_updater.erl
./src/couchdb/couch_work_queue.erl
./src/couchdb/priv/couch_js/http.c
./src/couchdb/priv/couch_js/http.h
./src/couchdb/priv/couch_js/main.c
./src/couchdb/priv/couch_js/utf8.c
./src/couchdb/priv/couch_js/utf8.h
./src/couchdb/priv/spawnkillable/couchspawnkillable.sh
./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
couch_rep:
./src/couchdb/couch_js_functions.hrl
./src/couchdb/couch_rep.erl
./src/couchdb/couch_rep_att.erl
./src/couchdb/couch_rep_changes_feed.erl
./src/couchdb/couch_rep_db_listener.erl
./src/couchdb/couch_rep_httpc.erl
./src/couchdb/couch_rep_missing_revs.erl
./src/couchdb/couch_rep_reader.erl
./src/couchdb/couch_rep_sup.erl
./src/couchdb/couch_rep_writer.erl
couch_externals:
./src/couchdb/couch_external_manager.erl
./src/couchdb/couch_external_server.erl
couch_httpd:
./src/couchdb/couch_httpd.erl
./src/couchdb/couch_httpd_auth.erl
./src/couchdb/couch_httpd_db.erl
./src/couchdb/couch_httpd_external.erl
./src/couchdb/couch_httpd_misc_handlers.erl
./src/couchdb/couch_httpd_oauth.erl
./src/couchdb/couch_httpd_rewrite.erl
./src/couchdb/couch_httpd_show.erl
./src/couchdb/couch_httpd_stats_handlers.erl
./src/couchdb/couch_httpd_vhost.erl
./src/couchdb/couch_httpd_view.erl
Re: Using rebar to install couchdb
Posted by Robert Dionne <di...@dionne-associates.com>.
+1
On Oct 13, 2010, at 5:23 PM, Benoit Chesneau wrote:
> In an attempt to start some merging with cloudant I would like to
> start by using rebar in our install process.
>
> Like i see it, we could continue to use autotools to create the
> rebar.config files and other templates an then rebar for the final
> build and dependencies management. This changes as noticed by @davisp
> also imply we make our tree a little more OTP compliant. I would like
> to start this work asap.
>
> Thoughts ?
>
> - benoit
Re: Using rebar to install couchdb
Posted by Juhani Ränkimies <ju...@juranki.com>.
Rebar isn't currently properly supported on windows, but making it
work should be doable. (http://bitbucket.org/juranki/rebar/ works for
me)
Using rebar might have potential to make it humane to work with
couchdb on windows.
+1
-juhani
On Thu, Oct 14, 2010 at 12:23 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> In an attempt to start some merging with cloudant I would like to
> start by using rebar in our install process.
>
> Like i see it, we could continue to use autotools to create the
> rebar.config files and other templates an then rebar for the final
> build and dependencies management. This changes as noticed by @davisp
> also imply we make our tree a little more OTP compliant. I would like
> to start this work asap.
>
> Thoughts ?
>
> - benoit
>