You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Adam Kocoloski <ko...@apache.org> on 2012/11/01 02:02:36 UTC
Source code layout
Hi, I mentioned in IRC earlier today that I'd really like to see us organize our source code so that OTP applications are wholly contained in their own subdirectories and are organized according to the standard OTP application layout. In a world where we've refactored the monolithic 'couch' application into a few more focused applications the layout could look something like this:
src/
couch_core/
c_src/
include/
priv/
src/
test/
couch_config/
src/
...
ibrowse/
include/
src/
test/
We've already followed this layout for couch_index, couch_mrview and couch_replicator -- I'd just like to "finish the job". A few of the advantages that I see are
a) We can extract these individual applications using e.g. git-subtree and use them on their own as we see fit.
b) We can remove the external dependencies (ibrowse, mochiweb) from our git repository and instead have the build system check them out directly from upstream. Of course they'd still show up directly in the release artefacts.
c) Rebar is the de facto build system for most other Erlang/OTP projects and it expects to see this kind of layout.
If there are no objections we can set about achieving this after we branch 1.3.x. Regards,
Adam
Re: Source code layout
Posted by Benoit Chesneau <bc...@gmail.com>.
What do you think about this proposal ? I would like to start some
work around soon.
On Thu, Nov 1, 2012 at 4:02 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>
>
>
> On Thu, Nov 1, 2012 at 12:39 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>
>> I see. So the enhanced layout is something we would do to better organize
>> couch_core. Aside from that you're still talking about following the
>> general layout that you've used with rcouch, right?
>>
>> Adam
>>
>
> still undecided if it's better to have a core composed of multiple apps in
> the repo or one app but nested folders in src like the inets thing, but yes.
>
> - benoît
Re: Source code layout
Posted by Benoit Chesneau <bc...@gmail.com>.
On Thu, Nov 1, 2012 at 12:39 PM, Adam Kocoloski <ko...@apache.org> wrote:
> I see. So the enhanced layout is something we would do to better organize
> couch_core. Aside from that you're still talking about following the
> general layout that you've used with rcouch, right?
>
> Adam
>
>
still undecided if it's better to have a core composed of multiple apps in
the repo or one app but nested folders in src like the inets thing, but yes.
- benoît
Re: Source code layout
Posted by Adam Kocoloski <ko...@apache.org>.
I see. So the enhanced layout is something we would do to better organize couch_core. Aside from that you're still talking about following the general layout that you've used with rcouch, right?
Adam
On Nov 1, 2012, at 4:46 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> I found the project I was thinking about: OTP
>
> https://github.com/erlang/otp/
>
> &
>
> https://github.com/erlang/otp/tree/maint/lib/inets/src
>
> inets look slike the couch_core structure I describe.
>
>
> - benoît
>
>
> On Thu, Nov 1, 2012 at 9:20 AM, Benoit Chesneau <bc...@gmail.com> wrote:
>
>> About the layout:
>> -------------------------
>>
>> So I did that work in rcouch:
>>
>> https://github.com/refuge/couch_core/tree/master/apps
>>
>> Each apps are self supervised. Then they are handled in the release:
>>
>> https://github.com/refuge/rcouch
>>
>> This is quite convenient to have it like this and pretty similar to what
>> you describe. It's in production since 9 months. Though I would extract
>> couch_mrview & couch_index from the repo (and put pack all_doc in core)
>> since m/r is not really the core even if they must be shipped in the
>> couchdb release. Probably the same with the couch_replicator app.
>>
>> It also allows to build custom releases:
>>
>> https://github.com/benoitc/rcouch_template
>>
>> Anyway, last day I found another pattern used at least on 2 projects (i
>> can't find them right now) that could be quite interresting:
>>
>> couch_core could be one app:
>>
>> c_src/
>> couchjs/
>> couch_collate/
>> include/
>> src/
>> couch.app.src
>> httpd/*.erl
>> core/*.erl
>> {mem3/ , ... }
>>
>> (not the second level can be skip if some are allergic to clean
>> separations using folders àla C)
>>
>>
>> Then we would have:
>>
>> couch_index/
>> src/
>> couch_index.app.src
>> *.erl
>> include/
>>
>> and so on
>>
>> Doing this would allows anyone building its own release to remove some
>> parts of the system while keep the couchdb core but would also provides
>> reall standalone Erlang applications that could be embedded in others
>> projects.
>>
>>
>> @davisp about the use of rebar vs autotools, I discussed it on a previous
>> mail but we could probably have the following scenatio:
>>
>> a bootstrap script building:
>> - a default rebar config for those of us who are only using rebar
>> - a configure script that will build a rebar.shared.config
>>
>> which gives in term of layout:
>>
>> /bootstrap
>> /configure.ac
>> /rebar.config.in
>>
>> ...
>>
>> It could be interresting to have a look in
>> https://github.com/okeuday/CloudI for that purpose.
>>
>> - benoît
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 1, 2012 at 2:02 AM, Adam Kocoloski <ko...@apache.org>wrote:
>>
>>> Hi, I mentioned in IRC earlier today that I'd really like to see us
>>> organize our source code so that OTP applications are wholly contained in
>>> their own subdirectories and are organized according to the standard OTP
>>> application layout. In a world where we've refactored the monolithic
>>> 'couch' application into a few more focused applications the layout could
>>> look something like this:
>>>
>>> src/
>>> couch_core/
>>> c_src/
>>> include/
>>> priv/
>>> src/
>>> test/
>>> couch_config/
>>> src/
>>> ...
>>> ibrowse/
>>> include/
>>> src/
>>> test/
>>>
>>> We've already followed this layout for couch_index, couch_mrview and
>>> couch_replicator -- I'd just like to "finish the job". A few of the
>>> advantages that I see are
>>>
>>> a) We can extract these individual applications using e.g. git-subtree
>>> and use them on their own as we see fit.
>>>
>>> b) We can remove the external dependencies (ibrowse, mochiweb) from our
>>> git repository and instead have the build system check them out directly
>>> from upstream. Of course they'd still show up directly in the release
>>> artefacts.
>>>
>>> c) Rebar is the de facto build system for most other Erlang/OTP projects
>>> and it expects to see this kind of layout.
>>>
>>> If there are no objections we can set about achieving this after we
>>> branch 1.3.x. Regards,
>>>
>>> Adam
>>
>>
>>
Re: Source code layout
Posted by Benoit Chesneau <bc...@gmail.com>.
I found the project I was thinking about: OTP
https://github.com/erlang/otp/
&
https://github.com/erlang/otp/tree/maint/lib/inets/src
inets look slike the couch_core structure I describe.
- benoît
On Thu, Nov 1, 2012 at 9:20 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> About the layout:
> -------------------------
>
> So I did that work in rcouch:
>
> https://github.com/refuge/couch_core/tree/master/apps
>
> Each apps are self supervised. Then they are handled in the release:
>
> https://github.com/refuge/rcouch
>
> This is quite convenient to have it like this and pretty similar to what
> you describe. It's in production since 9 months. Though I would extract
> couch_mrview & couch_index from the repo (and put pack all_doc in core)
> since m/r is not really the core even if they must be shipped in the
> couchdb release. Probably the same with the couch_replicator app.
>
> It also allows to build custom releases:
>
> https://github.com/benoitc/rcouch_template
>
> Anyway, last day I found another pattern used at least on 2 projects (i
> can't find them right now) that could be quite interresting:
>
> couch_core could be one app:
>
> c_src/
> couchjs/
> couch_collate/
> include/
> src/
> couch.app.src
> httpd/*.erl
> core/*.erl
> {mem3/ , ... }
>
> (not the second level can be skip if some are allergic to clean
> separations using folders àla C)
>
>
> Then we would have:
>
> couch_index/
> src/
> couch_index.app.src
> *.erl
> include/
>
> and so on
>
> Doing this would allows anyone building its own release to remove some
> parts of the system while keep the couchdb core but would also provides
> reall standalone Erlang applications that could be embedded in others
> projects.
>
>
> @davisp about the use of rebar vs autotools, I discussed it on a previous
> mail but we could probably have the following scenatio:
>
> a bootstrap script building:
> - a default rebar config for those of us who are only using rebar
> - a configure script that will build a rebar.shared.config
>
> which gives in term of layout:
>
> /bootstrap
> /configure.ac
> /rebar.config.in
>
> ...
>
> It could be interresting to have a look in
> https://github.com/okeuday/CloudI for that purpose.
>
> - benoît
>
>
>
>
>
>
> On Thu, Nov 1, 2012 at 2:02 AM, Adam Kocoloski <ko...@apache.org>wrote:
>
>> Hi, I mentioned in IRC earlier today that I'd really like to see us
>> organize our source code so that OTP applications are wholly contained in
>> their own subdirectories and are organized according to the standard OTP
>> application layout. In a world where we've refactored the monolithic
>> 'couch' application into a few more focused applications the layout could
>> look something like this:
>>
>> src/
>> couch_core/
>> c_src/
>> include/
>> priv/
>> src/
>> test/
>> couch_config/
>> src/
>> ...
>> ibrowse/
>> include/
>> src/
>> test/
>>
>> We've already followed this layout for couch_index, couch_mrview and
>> couch_replicator -- I'd just like to "finish the job". A few of the
>> advantages that I see are
>>
>> a) We can extract these individual applications using e.g. git-subtree
>> and use them on their own as we see fit.
>>
>> b) We can remove the external dependencies (ibrowse, mochiweb) from our
>> git repository and instead have the build system check them out directly
>> from upstream. Of course they'd still show up directly in the release
>> artefacts.
>>
>> c) Rebar is the de facto build system for most other Erlang/OTP projects
>> and it expects to see this kind of layout.
>>
>> If there are no objections we can set about achieving this after we
>> branch 1.3.x. Regards,
>>
>> Adam
>
>
>
Re: Source code layout
Posted by Benoit Chesneau <bc...@gmail.com>.
About the layout:
-------------------------
So I did that work in rcouch:
https://github.com/refuge/couch_core/tree/master/apps
Each apps are self supervised. Then they are handled in the release:
https://github.com/refuge/rcouch
This is quite convenient to have it like this and pretty similar to what
you describe. It's in production since 9 months. Though I would extract
couch_mrview & couch_index from the repo (and put pack all_doc in core)
since m/r is not really the core even if they must be shipped in the
couchdb release. Probably the same with the couch_replicator app.
It also allows to build custom releases:
https://github.com/benoitc/rcouch_template
Anyway, last day I found another pattern used at least on 2 projects (i
can't find them right now) that could be quite interresting:
couch_core could be one app:
c_src/
couchjs/
couch_collate/
include/
src/
couch.app.src
httpd/*.erl
core/*.erl
{mem3/ , ... }
(not the second level can be skip if some are allergic to clean separations
using folders àla C)
Then we would have:
couch_index/
src/
couch_index.app.src
*.erl
include/
and so on
Doing this would allows anyone building its own release to remove some
parts of the system while keep the couchdb core but would also provides
reall standalone Erlang applications that could be embedded in others
projects.
@davisp about the use of rebar vs autotools, I discussed it on a previous
mail but we could probably have the following scenatio:
a bootstrap script building:
- a default rebar config for those of us who are only using rebar
- a configure script that will build a rebar.shared.config
which gives in term of layout:
/bootstrap
/configure.ac
/rebar.config.in
...
It could be interresting to have a look in
https://github.com/okeuday/CloudIfor that purpose.
- benoît
On Thu, Nov 1, 2012 at 2:02 AM, Adam Kocoloski <ko...@apache.org> wrote:
> Hi, I mentioned in IRC earlier today that I'd really like to see us
> organize our source code so that OTP applications are wholly contained in
> their own subdirectories and are organized according to the standard OTP
> application layout. In a world where we've refactored the monolithic
> 'couch' application into a few more focused applications the layout could
> look something like this:
>
> src/
> couch_core/
> c_src/
> include/
> priv/
> src/
> test/
> couch_config/
> src/
> ...
> ibrowse/
> include/
> src/
> test/
>
> We've already followed this layout for couch_index, couch_mrview and
> couch_replicator -- I'd just like to "finish the job". A few of the
> advantages that I see are
>
> a) We can extract these individual applications using e.g. git-subtree and
> use them on their own as we see fit.
>
> b) We can remove the external dependencies (ibrowse, mochiweb) from our
> git repository and instead have the build system check them out directly
> from upstream. Of course they'd still show up directly in the release
> artefacts.
>
> c) Rebar is the de facto build system for most other Erlang/OTP projects
> and it expects to see this kind of layout.
>
> If there are no objections we can set about achieving this after we branch
> 1.3.x. Regards,
>
> Adam
Re: Source code layout
Posted by Benoit Chesneau <bc...@gmail.com>.
about multiple repo noah pointed yesterday on the cordova projet:
https://git-wip-us.apache.org/repos/asf?s=cordova
Not sure how each projects are considered though.
- benoît
On Thu, Nov 1, 2012 at 7:19 AM, Paul Davis <pa...@gmail.com>wrote:
> On Wed, Oct 31, 2012 at 9:02 PM, Adam Kocoloski <ko...@apache.org>
> wrote:
> > Hi, I mentioned in IRC earlier today that I'd really like to see us
> organize our source code so that OTP applications are wholly contained in
> their own subdirectories and are organized according to the standard OTP
> application layout. In a world where we've refactored the monolithic
> 'couch' application into a few more focused applications the layout could
> look something like this:
> >
> > src/
> > couch_core/
> > c_src/
> > include/
> > priv/
> > src/
> > test/
> > couch_config/
> > src/
> > ...
> > ibrowse/
> > include/
> > src/
> > test/
> >
> > We've already followed this layout for couch_index, couch_mrview and
> couch_replicator -- I'd just like to "finish the job". A few of the
> advantages that I see are
> >
> > a) We can extract these individual applications using e.g. git-subtree
> and use them on their own as we see fit.
> >
>
> I'd be fairly interested in seeing how git subtree works in real life.
> I've seen two rough uses of it. First, to include a project whole sale
> as a dependency. Ie, avoid requiring the build system to have to
> download a tarball and script that project's build system. The other
> is to do what you're suggesting where we manage them with subtree
> while their development history is tied more directly to the main
> project. I guess that's a really long winded way of "are we or are we
> not sending patches upstream as often as committing to master".
>
> For the "coordinated development" model which would require us to have
> multiple git repos for individual OTP apps we'll have to figure out if
> infra has relaxed their requirements on multiple git repos per TLP.
>
> > b) We can remove the external dependencies (ibrowse, mochiweb) from our
> git repository and instead have the build system check them out directly
> from upstream. Of course they'd still show up directly in the release
> artefacts.
> >
>
> This we actually can't do directly. We theoretically could have ASF
> mirrors like is common practice on GitHub (ie, fork a copy to own) but
> this comes with the caveats about multiple repos from A as well as
> probably blow back on why we're simply mirroring dependency projects.
>
> > c) Rebar is the de facto build system for most other Erlang/OTP projects
> and it expects to see this kind of layout.
> >
>
> While true, we should remember that we still haven't figured out a
> plan on how to actually use rebar with Autotools. Though I don't think
> this in anyway is an argument against splitting apps. I think its
> already pretty obvious that our code organization is terrible and we
> should be doing the source tree refactoring regardless. Just wanted to
> make a note.
>
> > If there are no objections we can set about achieving this after we
> branch 1.3.x. Regards,
> >
> > Adam
>
Re: Source code layout
Posted by Paul Davis <pa...@gmail.com>.
On Wed, Oct 31, 2012 at 9:02 PM, Adam Kocoloski <ko...@apache.org> wrote:
> Hi, I mentioned in IRC earlier today that I'd really like to see us organize our source code so that OTP applications are wholly contained in their own subdirectories and are organized according to the standard OTP application layout. In a world where we've refactored the monolithic 'couch' application into a few more focused applications the layout could look something like this:
>
> src/
> couch_core/
> c_src/
> include/
> priv/
> src/
> test/
> couch_config/
> src/
> ...
> ibrowse/
> include/
> src/
> test/
>
> We've already followed this layout for couch_index, couch_mrview and couch_replicator -- I'd just like to "finish the job". A few of the advantages that I see are
>
> a) We can extract these individual applications using e.g. git-subtree and use them on their own as we see fit.
>
I'd be fairly interested in seeing how git subtree works in real life.
I've seen two rough uses of it. First, to include a project whole sale
as a dependency. Ie, avoid requiring the build system to have to
download a tarball and script that project's build system. The other
is to do what you're suggesting where we manage them with subtree
while their development history is tied more directly to the main
project. I guess that's a really long winded way of "are we or are we
not sending patches upstream as often as committing to master".
For the "coordinated development" model which would require us to have
multiple git repos for individual OTP apps we'll have to figure out if
infra has relaxed their requirements on multiple git repos per TLP.
> b) We can remove the external dependencies (ibrowse, mochiweb) from our git repository and instead have the build system check them out directly from upstream. Of course they'd still show up directly in the release artefacts.
>
This we actually can't do directly. We theoretically could have ASF
mirrors like is common practice on GitHub (ie, fork a copy to own) but
this comes with the caveats about multiple repos from A as well as
probably blow back on why we're simply mirroring dependency projects.
> c) Rebar is the de facto build system for most other Erlang/OTP projects and it expects to see this kind of layout.
>
While true, we should remember that we still haven't figured out a
plan on how to actually use rebar with Autotools. Though I don't think
this in anyway is an argument against splitting apps. I think its
already pretty obvious that our code organization is terrible and we
should be doing the source tree refactoring regardless. Just wanted to
make a note.
> If there are no objections we can set about achieving this after we branch 1.3.x. Regards,
>
> Adam