You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Ilya Khlopotov <ii...@ca.ibm.com> on 2016/04/12 17:22:14 UTC
On dependency management and CI issues associated with it
Dear community,
There is a problem with contributors workflow which renders our CI system
useless. As you might know couchdb project consists of multiple
repositories. Most of the time changes cross the repositories boundaries.
When this happens the push to any of the repositories causes CI failures.
CI fails since it uses the old version of dependencies from main repository
of the project. Here is what we can do about it.
# Proposal
Let's use multiple files for dependency management.
- deps.json - serves the same purpose as dependencies list from current
rebar.config.script
- proposed.deps.json - here we specify list of PRs we want to commit
atomically
- override.deps.json - local file outside of version control which we
consult in order to include development tools specific to contributor (code
reloader, debugger, tracer, profiler, binpp, ...)
Bellow is the example of a content of these files:
## deps.json
{
"src/b64url": [
"https://github.com/apache/couchdb-b64url",
"6895652d80f95cdf04efb14625abed868998f174"
],
"src/cassim": [
"https://github.com/apache/couchdb-cassim",
"9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
],
"src/chttpd": [
"https://github.com/apache/couchdb-chttpd",
"54e8f6147486d9afc5245e0143d15a4dd1185654"
],
"src/meck": [
"https://github.com/apache/couchdb-meck",
"tree/0.8.2"
],
....
}
## proposed.deps.json
{
"src/couch": "https://github.com/apache/couchdb-couch/pull/124",
"src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"
"src/couch_tests": [
"https://github.com/apache/couchdb-erlang-tests",
"tree/branch"
],
}
# Interface
I propose to write a simple CLI tool to work with this structure. Bellow is
a list of commands which we need to support (for minimal version)
## Adding new dependency
git propose add https://github.com/apache/couchdb-foo
a2d5ad2eedc960248b806f61df0a1009462bdb46
git propose add https://github.com/apache/couchdb-bar tree/branch_name
## Adding new PR to the change set
git propose add https://github.com/apache/couchdb-config/pull/4
## Checking out right dependencies
git propose checkout
## Checking out release
git propose checkout --release # this would ignore proposed.deps.json if it
exists
## Merge the change
This command would do the following:
- Parse proposed.deps.json
- Retrieve merge commit sha for every PR (exit if dependency is not merged
yet)
- Update dependencies in deps.json with correct merge commit sha
- remove proposed.deps.json
# Workflow
export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv`
git checkout -b feature-ZZZ
cd src/X && hack dependency X
cd ../..
cd src/Y && hack dependency Y
issue PRs for X and Y
cd ../..
git propose add https://github.com/apache/couchdb-X/pull/4
git propose add https://github.com/apache/couchdb-Y/pull/49
git add proposed.deps.json
git commit -m "Commit feature {something} which does {a thing} and can be
tested as {procedure}"
git push origin feature-ZZZ
^ this would trigger our CI
CI would do
git propose checkout && ./configure && make check
# Pros and Cons
## Pros
- Changes are merged atomically
- CI runs against expected versions of deps
- Enables git bisect
- Reduce tasks that needs to be done by ASF committer (no need to update
rebar.config.script manually)
- Simplifies testing of PRs by reviewers
- Simplifies rebar.config since rebar is not used for managing deps
## Cons
- some github.com specifics (concept of PRs and access to github API to get
info about PR)
- we need to have github.com as one of the remotes
- we trigger CI only on push to main repository
# Implementation
We write a git-propose script in python and place it in ./bin. We add ./bin
into either GIT_EXEC_PATH or PATH. You always can call the script directly
(as ./bin/git-propose) if you don't like amending
your environment.
# Later improvements
- We can issue PRs from the tool itself
- We can merge from the tool itself
- We can implement support for multiple remotes (asf, github, private)
- We can implement support for multiple git transports (for the first
version all repositories in *.deps.json files would use https://)
Sincerely,
ILYA KHLOPOTOV
Re: On dependency management and CI issues associated with it
Posted by Paul Davis <pa...@gmail.com>.
I wouldn't quite go so far as to call it rude but it is at least
annoying that we're not part of the usptream network on GitHub.
Unfortunately, between ASF rules and regulations and the GitHub
integration there's not a whole lot we can do. Granted given that
we're not developing downstream in those repos I'm not sure its a big
deal to worry about. Any ASF committer that wants to make an upstream
change can fork the upstream repo, patch and PR, and then our
downstream mirrors can be updated.
On Wed, Apr 13, 2016 at 12:39 PM, Robert Newson <rn...@apache.org> wrote:
> I'd exclude third party repos, sure.
>
> It's a thread derail but this notion that we're being "fairly rude" needs resolving. It might be lost to history now but we got here, I think, with the best intentions of ensuring all the code that appears in couchdb can be traced back to code hosted at asf. Is it a concrete requirement? I honestly forget but I thought so.
>
> If it's not a requirement then we could point to non asf hosted repositories, if we accept that things we've used in the past could vanish at any time.
>
> A halfway might be a true fork hosted on GitHub? I think even we're not safer as the original project could be pulled and take the forks with it. This has happened. It's obviously unlikely.
>
>> On 13 Apr 2016, at 17:44, Adam Kocoloski <ko...@apache.org> wrote:
>>
>>
>>> On Apr 13, 2016, at 12:30 PM, Alexander Shorin <kx...@gmail.com> wrote:
>>>
>>> Hi Paul!
>>>
>>> Thanks for great input!
>>>
>>>> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis <pa...@gmail.com> wrote:
>>>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>>>> to hear it. Otherwise I may work up a lengthier and more thorough
>>>> proposal for dev@ to consider consolidating all of these repositories
>>>> for sanity and profit.
>>>
>>> It's hard to object against that since this actually solves a lot of
>>> problems solution of which will require more work to do and still will
>>> leave a place for mistakes or require quite specific toolchain to
>>> work.
>>>
>>> Making our current repos design work right will require even more work to apply.
>>>
>>> So, for point of time/resources/usability there is no much choice.
>>>
>>> I think folding the "Erlang repos developed by ASF" list will solve
>>> most part of the problems. However, I think it worth to keep these
>>> apps in own repos:
>>> - rexi
>>> - b64url
>>> - config
>>> - snappy
>>> - khash
>>> - ets-lru
>>> - twig (why we still need it?)
>>>
>>> As they could be reused outside and they shouldn't involve any
>>> dependencies with other couch modules by design. Everyone else may
>>> stand on where they are.
>>>
>>> P.S. I'm not sure if git-subtree will not introduce more new problems
>>> as it's quite tricky thing to live with it.
>>>
>>> --
>>> ,,,^..^,,,
>>
>> +1 Alex. I disagree that “one Erlang repo” is the goal. Erlang is just starting to get better support for package managers and the like, and it’d be a shame to miss out on opportunities for adoption of and contribution to general-purpose libraries like config, rexi, khash or ets-lru by bundling them into a repo that doesn’t look or feel like an idiomatic Erlang repo. And at any rate I certainly hope no one is proposing to copy/paste other upstream deps into one repo — the current practice of hosting a fork that doesn’t get recognized as a fork in GitHub is already fairly rude behavior on our part.
>>
>> I’ll agree that some of the repo creation has gotten a bit out of control. I believe _some_ of the friction is healthy; I think some of my contributions over the years were better precisely because the repo structure made bad designs harder to implement. The friction in the CI / CD pipeline and overall dependency management is not healthy, though.
>>
>> Moving more code back into one repo puts more responsibility on the devs to architect new enhancements in the right way without the repo structure there to enforce a few of the best practices. That’s OK so long as we understand the tradeoff.
>>
>> Adam
>>
>
Re: On dependency management and CI issues associated with it
Posted by Paul Davis <pa...@gmail.com>.
Ah, that sounds like a good approach to me.
On Fri, Apr 15, 2016 at 11:39 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
>> On 15 Apr 2016, at 14:38, Paul J Davis <pa...@gmail.com> wrote:
>>
>>
>>
>>> On Apr 15, 2016, at 3:13 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>>
>>>> On 14 Apr 2016, at 23:11, Joan Touzet <wo...@apache.org> wrote:
>>>>
>>>> Based on this information, are we in violation of ASF requirements? Can
>>>> anyone clarify for me what we actually need to be doing here?
>>>
>>> There is no such policy. We are also not bundling SpiderMonkey or Erlang
>>> either. Neither do any of the Java projects bundle e.g. OpenJDK.
>>>
>>
>> My understanding is that anything included in an ASF release tarball must be in ASF source control which is why we mirror mochiweb but not Erlang.
>
>> There are also rules about downloading things to build ASF projects and the Java/Maven communities should have this info. Given that Erlang hasn't had a real package thing until semi recently its not something I've spent time figuring out.
>
> Good subtlety! I remember a vigorous discussion about the maven stuff, we should check that out.
>
> In the meantime, this only makes things inconvenient as we’d prefer our end-users to having to run `npm install` on CouchDB install time. We can get “around” this by making the source tarball our Release, but recommend end-user recommend a tarball that includes the deps, but not make it that the official Release, like we do with the Mac build.
>
> Best
> Jan
> --
>
>
>>
>>> The question of whether to have a “safe copy“ to be ensured against
>>> suddenly disappearing upstream is entirely* up to the project, but not
>>> ASF policy.
>>>
>>> *upstream dependencies that have dual licensing that includes a GPL
>>> flavour or other incompatible license[1] can’t be mirrored on ASF
>>> source control and distribution servers (that’s why we don’t mirror
>>> SpiderMonkey or Erlang (although we could do Erlang now, that they
>>> switched to ASF 2, but I would not suggest we do this).
>>>
>>> [1]: http://www.apache.org/legal/resolved.html#category-x
>>>
>>> * * *
>>>
>>> Personally, with npm’s new unpublish policy[2], I’m okay with having
>>> our dependencies there.
>>>
>>> Because of the deep dependency tree, we should be very diligent about
>>> not accidentally including category-x licensed modules. I’m sure we
>>> can automate this into a npm postinstall script, so we know as soon
>>> as possible.
>>>
>>> At the very least, we need an audit prior to any release.
>>>
>>> [2]: http://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy
>>>
>>> Best
>>> Jan
>>> --
>>>
>>>
>>>>
>>>> -Joan
>>>>
>>>> ----- Original Message -----
>>>>> From: "Garren Smith" <ga...@apache.org>
>>>>> To: dev@couchdb.apache.org, "Joan Touzet" <wo...@apache.org>
>>>>> Sent: Thursday, April 14, 2016 2:43:10 AM
>>>>> Subject: Re: On dependency management and CI issues associated with it
>>>>>
>>>>> Hi Joan,
>>>>>
>>>>> Good point. Until about a week ago we use to keep all our
>>>>> dependencies in
>>>>> our repo. But we have just switched to webpack which allows us to
>>>>> manage
>>>>> our dependencies via npm (in case you are wondering, we don't depend
>>>>> on
>>>>> leftpad directly). So some of them are in our repo but the majority
>>>>> are
>>>>> downloaded and then bundled.
>>>>>
>>>>>
>>>>> Cheers
>>>>> Garren
>>>>>
>>>>> On Wed, Apr 13, 2016 at 11:29 PM, Joan Touzet <wo...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Garren, correct me if I'm wrong but Fauxton depends on a large
>>>>>> number
>>>>>> of JS dependencies that we don't keep copies of, correct? Or is it
>>>>>> just
>>>>>> for the build process?
>>>>>>
>>>>>> -Joan
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Alexander Shorin" <kx...@gmail.com>
>>>>>>> To: dev@couchdb.apache.org
>>>>>>> Sent: Wednesday, April 13, 2016 2:08:20 PM
>>>>>>> Subject: Re: On dependency management and CI issues associated
>>>>>>> with it
>>>>>>>
>>>>>>> On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson
>>>>>>> <rn...@apache.org>
>>>>>>> wrote:
>>>>>>>> It's a thread derail but this notion that we're being "fairly
>>>>>>>> rude"
>>>>>>>> needs resolving. It might be lost to history now but we got
>>>>>>>> here,
>>>>>>>> I think, with the best intentions of ensuring all the code that
>>>>>>>> appears in couchdb can be traced back to code hosted at asf. Is
>>>>>>>> it
>>>>>>>> a concrete requirement? I honestly forget but I thought so.
>>>>>>>
>>>>>>> Yes, that's the answer why. If one day mochiweb owner will decide
>>>>>>> to
>>>>>>> drop his github repo, we shouldn't be leave with broken builds.
>>>>>>> See
>>>>>>> leftpad story as well. Initially, that requirement seems
>>>>>>> redundant,
>>>>>>> but recent npm drama showed that it has a huge point. Also there
>>>>>>> are
>>>>>>> some legal bits about.
>>>>>>>
>>>>>>> --
>>>>>>> ,,,^..^,,,
>>>
>>> --
>>> Professional Support for Apache CouchDB:
>>> https://neighbourhood.ie/couchdb-support/
>>>
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
Re: On dependency management and CI issues associated with it
Posted by Jan Lehnardt <ja...@apache.org>.
> On 15 Apr 2016, at 14:38, Paul J Davis <pa...@gmail.com> wrote:
>
>
>
>> On Apr 15, 2016, at 3:13 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>
>>
>>> On 14 Apr 2016, at 23:11, Joan Touzet <wo...@apache.org> wrote:
>>>
>>> Based on this information, are we in violation of ASF requirements? Can
>>> anyone clarify for me what we actually need to be doing here?
>>
>> There is no such policy. We are also not bundling SpiderMonkey or Erlang
>> either. Neither do any of the Java projects bundle e.g. OpenJDK.
>>
>
> My understanding is that anything included in an ASF release tarball must be in ASF source control which is why we mirror mochiweb but not Erlang.
> There are also rules about downloading things to build ASF projects and the Java/Maven communities should have this info. Given that Erlang hasn't had a real package thing until semi recently its not something I've spent time figuring out.
Good subtlety! I remember a vigorous discussion about the maven stuff, we should check that out.
In the meantime, this only makes things inconvenient as we’d prefer our end-users to having to run `npm install` on CouchDB install time. We can get “around” this by making the source tarball our Release, but recommend end-user recommend a tarball that includes the deps, but not make it that the official Release, like we do with the Mac build.
Best
Jan
--
>
>> The question of whether to have a “safe copy“ to be ensured against
>> suddenly disappearing upstream is entirely* up to the project, but not
>> ASF policy.
>>
>> *upstream dependencies that have dual licensing that includes a GPL
>> flavour or other incompatible license[1] can’t be mirrored on ASF
>> source control and distribution servers (that’s why we don’t mirror
>> SpiderMonkey or Erlang (although we could do Erlang now, that they
>> switched to ASF 2, but I would not suggest we do this).
>>
>> [1]: http://www.apache.org/legal/resolved.html#category-x
>>
>> * * *
>>
>> Personally, with npm’s new unpublish policy[2], I’m okay with having
>> our dependencies there.
>>
>> Because of the deep dependency tree, we should be very diligent about
>> not accidentally including category-x licensed modules. I’m sure we
>> can automate this into a npm postinstall script, so we know as soon
>> as possible.
>>
>> At the very least, we need an audit prior to any release.
>>
>> [2]: http://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy
>>
>> Best
>> Jan
>> --
>>
>>
>>>
>>> -Joan
>>>
>>> ----- Original Message -----
>>>> From: "Garren Smith" <ga...@apache.org>
>>>> To: dev@couchdb.apache.org, "Joan Touzet" <wo...@apache.org>
>>>> Sent: Thursday, April 14, 2016 2:43:10 AM
>>>> Subject: Re: On dependency management and CI issues associated with it
>>>>
>>>> Hi Joan,
>>>>
>>>> Good point. Until about a week ago we use to keep all our
>>>> dependencies in
>>>> our repo. But we have just switched to webpack which allows us to
>>>> manage
>>>> our dependencies via npm (in case you are wondering, we don't depend
>>>> on
>>>> leftpad directly). So some of them are in our repo but the majority
>>>> are
>>>> downloaded and then bundled.
>>>>
>>>>
>>>> Cheers
>>>> Garren
>>>>
>>>> On Wed, Apr 13, 2016 at 11:29 PM, Joan Touzet <wo...@apache.org>
>>>> wrote:
>>>>
>>>>> Garren, correct me if I'm wrong but Fauxton depends on a large
>>>>> number
>>>>> of JS dependencies that we don't keep copies of, correct? Or is it
>>>>> just
>>>>> for the build process?
>>>>>
>>>>> -Joan
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Alexander Shorin" <kx...@gmail.com>
>>>>>> To: dev@couchdb.apache.org
>>>>>> Sent: Wednesday, April 13, 2016 2:08:20 PM
>>>>>> Subject: Re: On dependency management and CI issues associated
>>>>>> with it
>>>>>>
>>>>>> On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson
>>>>>> <rn...@apache.org>
>>>>>> wrote:
>>>>>>> It's a thread derail but this notion that we're being "fairly
>>>>>>> rude"
>>>>>>> needs resolving. It might be lost to history now but we got
>>>>>>> here,
>>>>>>> I think, with the best intentions of ensuring all the code that
>>>>>>> appears in couchdb can be traced back to code hosted at asf. Is
>>>>>>> it
>>>>>>> a concrete requirement? I honestly forget but I thought so.
>>>>>>
>>>>>> Yes, that's the answer why. If one day mochiweb owner will decide
>>>>>> to
>>>>>> drop his github repo, we shouldn't be leave with broken builds.
>>>>>> See
>>>>>> leftpad story as well. Initially, that requirement seems
>>>>>> redundant,
>>>>>> but recent npm drama showed that it has a huge point. Also there
>>>>>> are
>>>>>> some legal bits about.
>>>>>>
>>>>>> --
>>>>>> ,,,^..^,,,
>>
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: On dependency management and CI issues associated with it
Posted by Paul J Davis <pa...@gmail.com>.
> On Apr 15, 2016, at 3:13 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
>
>> On 14 Apr 2016, at 23:11, Joan Touzet <wo...@apache.org> wrote:
>>
>> Based on this information, are we in violation of ASF requirements? Can
>> anyone clarify for me what we actually need to be doing here?
>
> There is no such policy. We are also not bundling SpiderMonkey or Erlang
> either. Neither do any of the Java projects bundle e.g. OpenJDK.
>
My understanding is that anything included in an ASF release tarball must be in ASF source control which is why we mirror mochiweb but not Erlang.
There are also rules about downloading things to build ASF projects and the Java/Maven communities should have this info. Given that Erlang hasn't had a real package thing until semi recently its not something I've spent time figuring out.
> The question of whether to have a “safe copy“ to be ensured against
> suddenly disappearing upstream is entirely* up to the project, but not
> ASF policy.
>
> *upstream dependencies that have dual licensing that includes a GPL
> flavour or other incompatible license[1] can’t be mirrored on ASF
> source control and distribution servers (that’s why we don’t mirror
> SpiderMonkey or Erlang (although we could do Erlang now, that they
> switched to ASF 2, but I would not suggest we do this).
>
> [1]: http://www.apache.org/legal/resolved.html#category-x
>
> * * *
>
> Personally, with npm’s new unpublish policy[2], I’m okay with having
> our dependencies there.
>
> Because of the deep dependency tree, we should be very diligent about
> not accidentally including category-x licensed modules. I’m sure we
> can automate this into a npm postinstall script, so we know as soon
> as possible.
>
> At the very least, we need an audit prior to any release.
>
> [2]: http://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy
>
> Best
> Jan
> --
>
>
>>
>> -Joan
>>
>> ----- Original Message -----
>>> From: "Garren Smith" <ga...@apache.org>
>>> To: dev@couchdb.apache.org, "Joan Touzet" <wo...@apache.org>
>>> Sent: Thursday, April 14, 2016 2:43:10 AM
>>> Subject: Re: On dependency management and CI issues associated with it
>>>
>>> Hi Joan,
>>>
>>> Good point. Until about a week ago we use to keep all our
>>> dependencies in
>>> our repo. But we have just switched to webpack which allows us to
>>> manage
>>> our dependencies via npm (in case you are wondering, we don't depend
>>> on
>>> leftpad directly). So some of them are in our repo but the majority
>>> are
>>> downloaded and then bundled.
>>>
>>>
>>> Cheers
>>> Garren
>>>
>>> On Wed, Apr 13, 2016 at 11:29 PM, Joan Touzet <wo...@apache.org>
>>> wrote:
>>>
>>>> Garren, correct me if I'm wrong but Fauxton depends on a large
>>>> number
>>>> of JS dependencies that we don't keep copies of, correct? Or is it
>>>> just
>>>> for the build process?
>>>>
>>>> -Joan
>>>>
>>>> ----- Original Message -----
>>>>> From: "Alexander Shorin" <kx...@gmail.com>
>>>>> To: dev@couchdb.apache.org
>>>>> Sent: Wednesday, April 13, 2016 2:08:20 PM
>>>>> Subject: Re: On dependency management and CI issues associated
>>>>> with it
>>>>>
>>>>> On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson
>>>>> <rn...@apache.org>
>>>>> wrote:
>>>>>> It's a thread derail but this notion that we're being "fairly
>>>>>> rude"
>>>>>> needs resolving. It might be lost to history now but we got
>>>>>> here,
>>>>>> I think, with the best intentions of ensuring all the code that
>>>>>> appears in couchdb can be traced back to code hosted at asf. Is
>>>>>> it
>>>>>> a concrete requirement? I honestly forget but I thought so.
>>>>>
>>>>> Yes, that's the answer why. If one day mochiweb owner will decide
>>>>> to
>>>>> drop his github repo, we shouldn't be leave with broken builds.
>>>>> See
>>>>> leftpad story as well. Initially, that requirement seems
>>>>> redundant,
>>>>> but recent npm drama showed that it has a huge point. Also there
>>>>> are
>>>>> some legal bits about.
>>>>>
>>>>> --
>>>>> ,,,^..^,,,
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
Re: On dependency management and CI issues associated with it
Posted by Jan Lehnardt <ja...@apache.org>.
> On 14 Apr 2016, at 23:11, Joan Touzet <wo...@apache.org> wrote:
>
> Based on this information, are we in violation of ASF requirements? Can
> anyone clarify for me what we actually need to be doing here?
There is no such policy. We are also not bundling SpiderMonkey or Erlang
either. Neither do any of the Java projects bundle e.g. OpenJDK.
The question of whether to have a “safe copy“ to be ensured against
suddenly disappearing upstream is entirely* up to the project, but not
ASF policy.
*upstream dependencies that have dual licensing that includes a GPL
flavour or other incompatible license[1] can’t be mirrored on ASF
source control and distribution servers (that’s why we don’t mirror
SpiderMonkey or Erlang (although we could do Erlang now, that they
switched to ASF 2, but I would not suggest we do this).
[1]: http://www.apache.org/legal/resolved.html#category-x
* * *
Personally, with npm’s new unpublish policy[2], I’m okay with having
our dependencies there.
Because of the deep dependency tree, we should be very diligent about
not accidentally including category-x licensed modules. I’m sure we
can automate this into a npm postinstall script, so we know as soon
as possible.
At the very least, we need an audit prior to any release.
[2]: http://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy
Best
Jan
--
>
> -Joan
>
> ----- Original Message -----
>> From: "Garren Smith" <ga...@apache.org>
>> To: dev@couchdb.apache.org, "Joan Touzet" <wo...@apache.org>
>> Sent: Thursday, April 14, 2016 2:43:10 AM
>> Subject: Re: On dependency management and CI issues associated with it
>>
>> Hi Joan,
>>
>> Good point. Until about a week ago we use to keep all our
>> dependencies in
>> our repo. But we have just switched to webpack which allows us to
>> manage
>> our dependencies via npm (in case you are wondering, we don't depend
>> on
>> leftpad directly). So some of them are in our repo but the majority
>> are
>> downloaded and then bundled.
>>
>>
>> Cheers
>> Garren
>>
>> On Wed, Apr 13, 2016 at 11:29 PM, Joan Touzet <wo...@apache.org>
>> wrote:
>>
>>> Garren, correct me if I'm wrong but Fauxton depends on a large
>>> number
>>> of JS dependencies that we don't keep copies of, correct? Or is it
>>> just
>>> for the build process?
>>>
>>> -Joan
>>>
>>> ----- Original Message -----
>>>> From: "Alexander Shorin" <kx...@gmail.com>
>>>> To: dev@couchdb.apache.org
>>>> Sent: Wednesday, April 13, 2016 2:08:20 PM
>>>> Subject: Re: On dependency management and CI issues associated
>>>> with it
>>>>
>>>> On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson
>>>> <rn...@apache.org>
>>>> wrote:
>>>>> It's a thread derail but this notion that we're being "fairly
>>>>> rude"
>>>>> needs resolving. It might be lost to history now but we got
>>>>> here,
>>>>> I think, with the best intentions of ensuring all the code that
>>>>> appears in couchdb can be traced back to code hosted at asf. Is
>>>>> it
>>>>> a concrete requirement? I honestly forget but I thought so.
>>>>
>>>> Yes, that's the answer why. If one day mochiweb owner will decide
>>>> to
>>>> drop his github repo, we shouldn't be leave with broken builds.
>>>> See
>>>> leftpad story as well. Initially, that requirement seems
>>>> redundant,
>>>> but recent npm drama showed that it has a huge point. Also there
>>>> are
>>>> some legal bits about.
>>>>
>>>> --
>>>> ,,,^..^,,,
>>>>
>>>
>>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: On dependency management and CI issues associated with it
Posted by Joan Touzet <wo...@apache.org>.
Based on this information, are we in violation of ASF requirements? Can
anyone clarify for me what we actually need to be doing here?
-Joan
----- Original Message -----
> From: "Garren Smith" <ga...@apache.org>
> To: dev@couchdb.apache.org, "Joan Touzet" <wo...@apache.org>
> Sent: Thursday, April 14, 2016 2:43:10 AM
> Subject: Re: On dependency management and CI issues associated with it
>
> Hi Joan,
>
> Good point. Until about a week ago we use to keep all our
> dependencies in
> our repo. But we have just switched to webpack which allows us to
> manage
> our dependencies via npm (in case you are wondering, we don't depend
> on
> leftpad directly). So some of them are in our repo but the majority
> are
> downloaded and then bundled.
>
>
> Cheers
> Garren
>
> On Wed, Apr 13, 2016 at 11:29 PM, Joan Touzet <wo...@apache.org>
> wrote:
>
> > Garren, correct me if I'm wrong but Fauxton depends on a large
> > number
> > of JS dependencies that we don't keep copies of, correct? Or is it
> > just
> > for the build process?
> >
> > -Joan
> >
> > ----- Original Message -----
> > > From: "Alexander Shorin" <kx...@gmail.com>
> > > To: dev@couchdb.apache.org
> > > Sent: Wednesday, April 13, 2016 2:08:20 PM
> > > Subject: Re: On dependency management and CI issues associated
> > > with it
> > >
> > > On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson
> > > <rn...@apache.org>
> > > wrote:
> > > > It's a thread derail but this notion that we're being "fairly
> > > > rude"
> > > > needs resolving. It might be lost to history now but we got
> > > > here,
> > > > I think, with the best intentions of ensuring all the code that
> > > > appears in couchdb can be traced back to code hosted at asf. Is
> > > > it
> > > > a concrete requirement? I honestly forget but I thought so.
> > >
> > > Yes, that's the answer why. If one day mochiweb owner will decide
> > > to
> > > drop his github repo, we shouldn't be leave with broken builds.
> > > See
> > > leftpad story as well. Initially, that requirement seems
> > > redundant,
> > > but recent npm drama showed that it has a huge point. Also there
> > > are
> > > some legal bits about.
> > >
> > > --
> > > ,,,^..^,,,
> > >
> >
>
Re: On dependency management and CI issues associated with it
Posted by Garren Smith <ga...@apache.org>.
Hi Joan,
Good point. Until about a week ago we use to keep all our dependencies in
our repo. But we have just switched to webpack which allows us to manage
our dependencies via npm (in case you are wondering, we don't depend on
leftpad directly). So some of them are in our repo but the majority are
downloaded and then bundled.
Cheers
Garren
On Wed, Apr 13, 2016 at 11:29 PM, Joan Touzet <wo...@apache.org> wrote:
> Garren, correct me if I'm wrong but Fauxton depends on a large number
> of JS dependencies that we don't keep copies of, correct? Or is it just
> for the build process?
>
> -Joan
>
> ----- Original Message -----
> > From: "Alexander Shorin" <kx...@gmail.com>
> > To: dev@couchdb.apache.org
> > Sent: Wednesday, April 13, 2016 2:08:20 PM
> > Subject: Re: On dependency management and CI issues associated with it
> >
> > On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson <rn...@apache.org>
> > wrote:
> > > It's a thread derail but this notion that we're being "fairly rude"
> > > needs resolving. It might be lost to history now but we got here,
> > > I think, with the best intentions of ensuring all the code that
> > > appears in couchdb can be traced back to code hosted at asf. Is it
> > > a concrete requirement? I honestly forget but I thought so.
> >
> > Yes, that's the answer why. If one day mochiweb owner will decide to
> > drop his github repo, we shouldn't be leave with broken builds. See
> > leftpad story as well. Initially, that requirement seems redundant,
> > but recent npm drama showed that it has a huge point. Also there are
> > some legal bits about.
> >
> > --
> > ,,,^..^,,,
> >
>
Re: On dependency management and CI issues associated with it
Posted by Adam Kocoloski <ko...@apache.org>.
> On Apr 13, 2016, at 2:08 PM, Alexander Shorin <kx...@gmail.com> wrote:
>
> On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson <rn...@apache.org> wrote:
>> It's a thread derail but this notion that we're being "fairly rude" needs resolving. It might be lost to history now but we got here, I think, with the best intentions of ensuring all the code that appears in couchdb can be traced back to code hosted at asf. Is it a concrete requirement? I honestly forget but I thought so.
>
> Yes, that's the answer why. If one day mochiweb owner will decide to
> drop his github repo, we shouldn't be leave with broken builds. See
> leftpad story as well. Initially, that requirement seems redundant,
> but recent npm drama showed that it has a huge point. Also there are
> some legal bits about.
>
> --
> ,,,^..^,,,
All good points. I’m fine running off a fork but would like to see Apache give the author credit in the GitHub network.
Adam
Re: On dependency management and CI issues associated with it
Posted by Joan Touzet <wo...@apache.org>.
Garren, correct me if I'm wrong but Fauxton depends on a large number
of JS dependencies that we don't keep copies of, correct? Or is it just
for the build process?
-Joan
----- Original Message -----
> From: "Alexander Shorin" <kx...@gmail.com>
> To: dev@couchdb.apache.org
> Sent: Wednesday, April 13, 2016 2:08:20 PM
> Subject: Re: On dependency management and CI issues associated with it
>
> On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson <rn...@apache.org>
> wrote:
> > It's a thread derail but this notion that we're being "fairly rude"
> > needs resolving. It might be lost to history now but we got here,
> > I think, with the best intentions of ensuring all the code that
> > appears in couchdb can be traced back to code hosted at asf. Is it
> > a concrete requirement? I honestly forget but I thought so.
>
> Yes, that's the answer why. If one day mochiweb owner will decide to
> drop his github repo, we shouldn't be leave with broken builds. See
> leftpad story as well. Initially, that requirement seems redundant,
> but recent npm drama showed that it has a huge point. Also there are
> some legal bits about.
>
> --
> ,,,^..^,,,
>
Re: On dependency management and CI issues associated with it
Posted by Alexander Shorin <kx...@gmail.com>.
On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson <rn...@apache.org> wrote:
> It's a thread derail but this notion that we're being "fairly rude" needs resolving. It might be lost to history now but we got here, I think, with the best intentions of ensuring all the code that appears in couchdb can be traced back to code hosted at asf. Is it a concrete requirement? I honestly forget but I thought so.
Yes, that's the answer why. If one day mochiweb owner will decide to
drop his github repo, we shouldn't be leave with broken builds. See
leftpad story as well. Initially, that requirement seems redundant,
but recent npm drama showed that it has a huge point. Also there are
some legal bits about.
--
,,,^..^,,,
Re: On dependency management and CI issues associated with it
Posted by Robert Newson <rn...@apache.org>.
I'd exclude third party repos, sure.
It's a thread derail but this notion that we're being "fairly rude" needs resolving. It might be lost to history now but we got here, I think, with the best intentions of ensuring all the code that appears in couchdb can be traced back to code hosted at asf. Is it a concrete requirement? I honestly forget but I thought so.
If it's not a requirement then we could point to non asf hosted repositories, if we accept that things we've used in the past could vanish at any time.
A halfway might be a true fork hosted on GitHub? I think even we're not safer as the original project could be pulled and take the forks with it. This has happened. It's obviously unlikely.
> On 13 Apr 2016, at 17:44, Adam Kocoloski <ko...@apache.org> wrote:
>
>
>> On Apr 13, 2016, at 12:30 PM, Alexander Shorin <kx...@gmail.com> wrote:
>>
>> Hi Paul!
>>
>> Thanks for great input!
>>
>>> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis <pa...@gmail.com> wrote:
>>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>>> to hear it. Otherwise I may work up a lengthier and more thorough
>>> proposal for dev@ to consider consolidating all of these repositories
>>> for sanity and profit.
>>
>> It's hard to object against that since this actually solves a lot of
>> problems solution of which will require more work to do and still will
>> leave a place for mistakes or require quite specific toolchain to
>> work.
>>
>> Making our current repos design work right will require even more work to apply.
>>
>> So, for point of time/resources/usability there is no much choice.
>>
>> I think folding the "Erlang repos developed by ASF" list will solve
>> most part of the problems. However, I think it worth to keep these
>> apps in own repos:
>> - rexi
>> - b64url
>> - config
>> - snappy
>> - khash
>> - ets-lru
>> - twig (why we still need it?)
>>
>> As they could be reused outside and they shouldn't involve any
>> dependencies with other couch modules by design. Everyone else may
>> stand on where they are.
>>
>> P.S. I'm not sure if git-subtree will not introduce more new problems
>> as it's quite tricky thing to live with it.
>>
>> --
>> ,,,^..^,,,
>
> +1 Alex. I disagree that “one Erlang repo” is the goal. Erlang is just starting to get better support for package managers and the like, and it’d be a shame to miss out on opportunities for adoption of and contribution to general-purpose libraries like config, rexi, khash or ets-lru by bundling them into a repo that doesn’t look or feel like an idiomatic Erlang repo. And at any rate I certainly hope no one is proposing to copy/paste other upstream deps into one repo — the current practice of hosting a fork that doesn’t get recognized as a fork in GitHub is already fairly rude behavior on our part.
>
> I’ll agree that some of the repo creation has gotten a bit out of control. I believe _some_ of the friction is healthy; I think some of my contributions over the years were better precisely because the repo structure made bad designs harder to implement. The friction in the CI / CD pipeline and overall dependency management is not healthy, though.
>
> Moving more code back into one repo puts more responsibility on the devs to architect new enhancements in the right way without the repo structure there to enforce a few of the best practices. That’s OK so long as we understand the tradeoff.
>
> Adam
>
Re: On dependency management and CI issues associated with it
Posted by Alexander Shorin <kx...@gmail.com>.
On Tue, Apr 19, 2016 at 11:26 PM, Ilya Khlopotov <ii...@ca.ibm.com> wrote:
>
> Thank you for you feedback and great discussion. It looks like I am the only one who prefer multi-repo model (with tools assistance).
You're not alone, actually, but it's hard to fight with reality now. I
think we all agreed that multi-repo is good, but at some point of
decomposition all the profits get covered by growing number of
problems: infrastructure, tools, learning curve, workflow.
Actually, I did a notice to myself to look into how other projects has
same repos model and how they deal with it. I don't think we're unique
on this problem.
If we'll get the good solution for most of our current issues, we can
stay with what we have. I don't think there is somebody who would love
to fold the repos preserving history and else bits. But so far...
--
,,,^..^,,,
Re: On dependency management and CI issues associated with it
Posted by Ilya Khlopotov <ii...@ca.ibm.com>.
Hey all,
Thank you for you feedback and great discussion. It looks like I am the
only one who prefer multi-repo model (with tools assistance).
BR,
ILYA
From: Jan Lehnardt <ja...@apache.org>
To: dev@couchdb.apache.org
Date: 2016/04/15 01:39 AM
Subject: Re: On dependency management and CI issues associated with it
Hey all, love the discussion! :)
I’ve identified these issues in this thread:
1. multi-repo vs. mono-repo
- with the sub-issues of how mono the repo should be
- and how to get tooling and process going specifically
2. keeping safe copies of upstream dependencies to avoid left-padding
- with the sub-issue of not being able to do this for every single one
because of licensing.
Let’s try to keep those separate :)
* * * Summary
As for 1, I think we have a rough consensus about going back to some form
of mono-repo. I don’t think anyone disagrees with keeping fauxton and the
docs in separate repos as well, same with nano and nmo, essentially
everything “non-core Erlang”.
The open question then is do we bundle *everything* else together,
including
the Erlang apps that are usable standalone as outlined by Paul, or fold
them
in. I think we have to further separate these by “developed by us” and
“forks
from upstream“ like mochiweb, ibrowse and jiffy (although, nice overlap
here :).
2. we should be discussing. Which deps are we afraid of going away, what
should
our policy be, etc? But let’s spin this out in a new thread. If you want to
reply to this, please choose a new Subject line.
* * * Opinion
From a “this sounds neat”-perspective, I’d prefer these three tiers:
1. one repo for all ASF-developed CouchDB bits that are not reasonably
different
Erlang apps (couch-replicator, chttpd, etc)*
2. one repo each for ASF-developed CouchDB bits that are reasonably used in
other projects (khash, b64url, snappy, etc)
3. one repo each for upstream forks.
* although they should continue to be separated in src/subdirs, and not all
in
src/couchdb as we did in 1.x.
My assumption is that category 2 deps are usually not the ones causing
trouble
in today’s development flow (please correct me if I’m wrong).
And I don’t know if development tooling gets any easier if we keep some
things
separate and others not, but if at all possible, I think the above is the
most
sensible.
I don’t have too much experience with git subtree, but the docs are surely
not beginner friendly, but I’m sure with a nice dev guide, we should be
able
to follow along.
Best
Jan
--
> On 13 Apr 2016, at 19:42, Robert Newson <rn...@apache.org> wrote:
>
> As for the separation we have enforcing good practices, I don't buy it.
>
> I don't think it will be difficult to prevent the kind of coupling you
(and I) would find troubling. It might even be easier to see if a single
commit touches multiple src/ subdirectories that might be missed when
reviewing separate pr's. At least, I'm not sure I could slide a credit card
between them.
>
>> On 13 Apr 2016, at 17:44, Adam Kocoloski <ko...@apache.org> wrote:
>>
>>
>>> On Apr 13, 2016, at 12:30 PM, Alexander Shorin <kx...@gmail.com>
wrote:
>>>
>>> Hi Paul!
>>>
>>> Thanks for great input!
>>>
>>>> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis
<pa...@gmail.com> wrote:
>>>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>>>> to hear it. Otherwise I may work up a lengthier and more thorough
>>>> proposal for dev@ to consider consolidating all of these repositories
>>>> for sanity and profit.
>>>
>>> It's hard to object against that since this actually solves a lot of
>>> problems solution of which will require more work to do and still will
>>> leave a place for mistakes or require quite specific toolchain to
>>> work.
>>>
>>> Making our current repos design work right will require even more work
to apply.
>>>
>>> So, for point of time/resources/usability there is no much choice.
>>>
>>> I think folding the "Erlang repos developed by ASF" list will solve
>>> most part of the problems. However, I think it worth to keep these
>>> apps in own repos:
>>> - rexi
>>> - b64url
>>> - config
>>> - snappy
>>> - khash
>>> - ets-lru
>>> - twig (why we still need it?)
>>>
>>> As they could be reused outside and they shouldn't involve any
>>> dependencies with other couch modules by design. Everyone else may
>>> stand on where they are.
>>>
>>> P.S. I'm not sure if git-subtree will not introduce more new problems
>>> as it's quite tricky thing to live with it.
>>>
>>> --
>>> ,,,^..^,,,
>>
>> +1 Alex. I disagree that “one Erlang repo” is the goal. Erlang is just
starting to get better support for package managers and the like, and it’d
be a shame to miss out on opportunities for adoption of and contribution to
general-purpose libraries like config, rexi, khash or ets-lru by bundling
them into a repo that doesn’t look or feel like an idiomatic Erlang repo.
And at any rate I certainly hope no one is proposing to copy/paste other
upstream deps into one repo — the current practice of hosting a fork that
doesn’t get recognized as a fork in GitHub is already fairly rude behavior
on our part.
>>
>> I’ll agree that some of the repo creation has gotten a bit out of
control. I believe _some_ of the friction is healthy; I think some of my
contributions over the years were better precisely because the repo
structure made bad designs harder to implement. The friction in the CI / CD
pipeline and overall dependency management is not healthy, though.
>>
>> Moving more code back into one repo puts more responsibility on the devs
to architect new enhancements in the right way without the repo structure
there to enforce a few of the best practices. That’s OK so long as we
understand the tradeoff.
>>
>> Adam
>>
>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: On dependency management and CI issues associated with it
Posted by Jan Lehnardt <ja...@apache.org>.
Hey all, love the discussion! :)
I’ve identified these issues in this thread:
1. multi-repo vs. mono-repo
- with the sub-issues of how mono the repo should be
- and how to get tooling and process going specifically
2. keeping safe copies of upstream dependencies to avoid left-padding
- with the sub-issue of not being able to do this for every single one
because of licensing.
Let’s try to keep those separate :)
* * * Summary
As for 1, I think we have a rough consensus about going back to some form
of mono-repo. I don’t think anyone disagrees with keeping fauxton and the
docs in separate repos as well, same with nano and nmo, essentially
everything “non-core Erlang”.
The open question then is do we bundle *everything* else together, including
the Erlang apps that are usable standalone as outlined by Paul, or fold them
in. I think we have to further separate these by “developed by us” and “forks
from upstream“ like mochiweb, ibrowse and jiffy (although, nice overlap here :).
2. we should be discussing. Which deps are we afraid of going away, what should
our policy be, etc? But let’s spin this out in a new thread. If you want to
reply to this, please choose a new Subject line.
* * * Opinion
From a “this sounds neat”-perspective, I’d prefer these three tiers:
1. one repo for all ASF-developed CouchDB bits that are not reasonably different
Erlang apps (couch-replicator, chttpd, etc)*
2. one repo each for ASF-developed CouchDB bits that are reasonably used in
other projects (khash, b64url, snappy, etc)
3. one repo each for upstream forks.
* although they should continue to be separated in src/subdirs, and not all in
src/couchdb as we did in 1.x.
My assumption is that category 2 deps are usually not the ones causing trouble
in today’s development flow (please correct me if I’m wrong).
And I don’t know if development tooling gets any easier if we keep some things
separate and others not, but if at all possible, I think the above is the most
sensible.
I don’t have too much experience with git subtree, but the docs are surely
not beginner friendly, but I’m sure with a nice dev guide, we should be able
to follow along.
Best
Jan
--
> On 13 Apr 2016, at 19:42, Robert Newson <rn...@apache.org> wrote:
>
> As for the separation we have enforcing good practices, I don't buy it.
>
> I don't think it will be difficult to prevent the kind of coupling you (and I) would find troubling. It might even be easier to see if a single commit touches multiple src/ subdirectories that might be missed when reviewing separate pr's. At least, I'm not sure I could slide a credit card between them.
>
>> On 13 Apr 2016, at 17:44, Adam Kocoloski <ko...@apache.org> wrote:
>>
>>
>>> On Apr 13, 2016, at 12:30 PM, Alexander Shorin <kx...@gmail.com> wrote:
>>>
>>> Hi Paul!
>>>
>>> Thanks for great input!
>>>
>>>> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis <pa...@gmail.com> wrote:
>>>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>>>> to hear it. Otherwise I may work up a lengthier and more thorough
>>>> proposal for dev@ to consider consolidating all of these repositories
>>>> for sanity and profit.
>>>
>>> It's hard to object against that since this actually solves a lot of
>>> problems solution of which will require more work to do and still will
>>> leave a place for mistakes or require quite specific toolchain to
>>> work.
>>>
>>> Making our current repos design work right will require even more work to apply.
>>>
>>> So, for point of time/resources/usability there is no much choice.
>>>
>>> I think folding the "Erlang repos developed by ASF" list will solve
>>> most part of the problems. However, I think it worth to keep these
>>> apps in own repos:
>>> - rexi
>>> - b64url
>>> - config
>>> - snappy
>>> - khash
>>> - ets-lru
>>> - twig (why we still need it?)
>>>
>>> As they could be reused outside and they shouldn't involve any
>>> dependencies with other couch modules by design. Everyone else may
>>> stand on where they are.
>>>
>>> P.S. I'm not sure if git-subtree will not introduce more new problems
>>> as it's quite tricky thing to live with it.
>>>
>>> --
>>> ,,,^..^,,,
>>
>> +1 Alex. I disagree that “one Erlang repo” is the goal. Erlang is just starting to get better support for package managers and the like, and it’d be a shame to miss out on opportunities for adoption of and contribution to general-purpose libraries like config, rexi, khash or ets-lru by bundling them into a repo that doesn’t look or feel like an idiomatic Erlang repo. And at any rate I certainly hope no one is proposing to copy/paste other upstream deps into one repo — the current practice of hosting a fork that doesn’t get recognized as a fork in GitHub is already fairly rude behavior on our part.
>>
>> I’ll agree that some of the repo creation has gotten a bit out of control. I believe _some_ of the friction is healthy; I think some of my contributions over the years were better precisely because the repo structure made bad designs harder to implement. The friction in the CI / CD pipeline and overall dependency management is not healthy, though.
>>
>> Moving more code back into one repo puts more responsibility on the devs to architect new enhancements in the right way without the repo structure there to enforce a few of the best practices. That’s OK so long as we understand the tradeoff.
>>
>> Adam
>>
>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: On dependency management and CI issues associated with it
Posted by Robert Newson <rn...@apache.org>.
As for the separation we have enforcing good practices, I don't buy it.
I don't think it will be difficult to prevent the kind of coupling you (and I) would find troubling. It might even be easier to see if a single commit touches multiple src/ subdirectories that might be missed when reviewing separate pr's. At least, I'm not sure I could slide a credit card between them.
> On 13 Apr 2016, at 17:44, Adam Kocoloski <ko...@apache.org> wrote:
>
>
>> On Apr 13, 2016, at 12:30 PM, Alexander Shorin <kx...@gmail.com> wrote:
>>
>> Hi Paul!
>>
>> Thanks for great input!
>>
>>> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis <pa...@gmail.com> wrote:
>>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>>> to hear it. Otherwise I may work up a lengthier and more thorough
>>> proposal for dev@ to consider consolidating all of these repositories
>>> for sanity and profit.
>>
>> It's hard to object against that since this actually solves a lot of
>> problems solution of which will require more work to do and still will
>> leave a place for mistakes or require quite specific toolchain to
>> work.
>>
>> Making our current repos design work right will require even more work to apply.
>>
>> So, for point of time/resources/usability there is no much choice.
>>
>> I think folding the "Erlang repos developed by ASF" list will solve
>> most part of the problems. However, I think it worth to keep these
>> apps in own repos:
>> - rexi
>> - b64url
>> - config
>> - snappy
>> - khash
>> - ets-lru
>> - twig (why we still need it?)
>>
>> As they could be reused outside and they shouldn't involve any
>> dependencies with other couch modules by design. Everyone else may
>> stand on where they are.
>>
>> P.S. I'm not sure if git-subtree will not introduce more new problems
>> as it's quite tricky thing to live with it.
>>
>> --
>> ,,,^..^,,,
>
> +1 Alex. I disagree that “one Erlang repo” is the goal. Erlang is just starting to get better support for package managers and the like, and it’d be a shame to miss out on opportunities for adoption of and contribution to general-purpose libraries like config, rexi, khash or ets-lru by bundling them into a repo that doesn’t look or feel like an idiomatic Erlang repo. And at any rate I certainly hope no one is proposing to copy/paste other upstream deps into one repo — the current practice of hosting a fork that doesn’t get recognized as a fork in GitHub is already fairly rude behavior on our part.
>
> I’ll agree that some of the repo creation has gotten a bit out of control. I believe _some_ of the friction is healthy; I think some of my contributions over the years were better precisely because the repo structure made bad designs harder to implement. The friction in the CI / CD pipeline and overall dependency management is not healthy, though.
>
> Moving more code back into one repo puts more responsibility on the devs to architect new enhancements in the right way without the repo structure there to enforce a few of the best practices. That’s OK so long as we understand the tradeoff.
>
> Adam
>
Re: On dependency management and CI issues associated with it
Posted by Adam Kocoloski <ko...@apache.org>.
> On Apr 13, 2016, at 12:30 PM, Alexander Shorin <kx...@gmail.com> wrote:
>
> Hi Paul!
>
> Thanks for great input!
>
> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis <pa...@gmail.com> wrote:
>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>> to hear it. Otherwise I may work up a lengthier and more thorough
>> proposal for dev@ to consider consolidating all of these repositories
>> for sanity and profit.
>
> It's hard to object against that since this actually solves a lot of
> problems solution of which will require more work to do and still will
> leave a place for mistakes or require quite specific toolchain to
> work.
>
> Making our current repos design work right will require even more work to apply.
>
> So, for point of time/resources/usability there is no much choice.
>
> I think folding the "Erlang repos developed by ASF" list will solve
> most part of the problems. However, I think it worth to keep these
> apps in own repos:
> - rexi
> - b64url
> - config
> - snappy
> - khash
> - ets-lru
> - twig (why we still need it?)
>
> As they could be reused outside and they shouldn't involve any
> dependencies with other couch modules by design. Everyone else may
> stand on where they are.
>
> P.S. I'm not sure if git-subtree will not introduce more new problems
> as it's quite tricky thing to live with it.
>
> --
> ,,,^..^,,,
+1 Alex. I disagree that “one Erlang repo” is the goal. Erlang is just starting to get better support for package managers and the like, and it’d be a shame to miss out on opportunities for adoption of and contribution to general-purpose libraries like config, rexi, khash or ets-lru by bundling them into a repo that doesn’t look or feel like an idiomatic Erlang repo. And at any rate I certainly hope no one is proposing to copy/paste other upstream deps into one repo — the current practice of hosting a fork that doesn’t get recognized as a fork in GitHub is already fairly rude behavior on our part.
I’ll agree that some of the repo creation has gotten a bit out of control. I believe _some_ of the friction is healthy; I think some of my contributions over the years were better precisely because the repo structure made bad designs harder to implement. The friction in the CI / CD pipeline and overall dependency management is not healthy, though.
Moving more code back into one repo puts more responsibility on the devs to architect new enhancements in the right way without the repo structure there to enforce a few of the best practices. That’s OK so long as we understand the tradeoff.
Adam
Re: On dependency management and CI issues associated with it
Posted by Nick Vatamaniuc <va...@gmail.com>.
No experience using subtrees, but remember Rust switched to use those:
https://github.com/rust-lang/rust/tree/master/src
jemalloc, llvm and few others are subtrees.
PR with some discussion: https://github.com/rust-lang/rust/pull/26042
-Nick
On Wed, Apr 13, 2016 at 12:38 PM, Paul Davis
<pa...@gmail.com> wrote:
> Does anyone have any experience with git subtree on this list? I was
> under the impression that as long as you ensured that it was a strict
> copy of upstream it was fairly simple.
>
> For your list of repos to keep separate, those sound fine to me
> regardless of subtree. They're all fairly stable at this point and I
> agree at least plausibly usable outside CouchDB.
>
> On Wed, Apr 13, 2016 at 11:30 AM, Alexander Shorin <kx...@gmail.com> wrote:
>> Hi Paul!
>>
>> Thanks for great input!
>>
>> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis <pa...@gmail.com> wrote:
>>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>>> to hear it. Otherwise I may work up a lengthier and more thorough
>>> proposal for dev@ to consider consolidating all of these repositories
>>> for sanity and profit.
>>
>> It's hard to object against that since this actually solves a lot of
>> problems solution of which will require more work to do and still will
>> leave a place for mistakes or require quite specific toolchain to
>> work.
>>
>> Making our current repos design work right will require even more work to apply.
>>
>> So, for point of time/resources/usability there is no much choice.
>>
>> I think folding the "Erlang repos developed by ASF" list will solve
>> most part of the problems. However, I think it worth to keep these
>> apps in own repos:
>> - rexi
>> - b64url
>> - config
>> - snappy
>> - khash
>> - ets-lru
>> - twig (why we still need it?)
>>
>> As they could be reused outside and they shouldn't involve any
>> dependencies with other couch modules by design. Everyone else may
>> stand on where they are.
>>
>> P.S. I'm not sure if git-subtree will not introduce more new problems
>> as it's quite tricky thing to live with it.
>>
>> --
>> ,,,^..^,,,
Re: On dependency management and CI issues associated with it
Posted by Paul Davis <pa...@gmail.com>.
Does anyone have any experience with git subtree on this list? I was
under the impression that as long as you ensured that it was a strict
copy of upstream it was fairly simple.
For your list of repos to keep separate, those sound fine to me
regardless of subtree. They're all fairly stable at this point and I
agree at least plausibly usable outside CouchDB.
On Wed, Apr 13, 2016 at 11:30 AM, Alexander Shorin <kx...@gmail.com> wrote:
> Hi Paul!
>
> Thanks for great input!
>
> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis <pa...@gmail.com> wrote:
>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>> to hear it. Otherwise I may work up a lengthier and more thorough
>> proposal for dev@ to consider consolidating all of these repositories
>> for sanity and profit.
>
> It's hard to object against that since this actually solves a lot of
> problems solution of which will require more work to do and still will
> leave a place for mistakes or require quite specific toolchain to
> work.
>
> Making our current repos design work right will require even more work to apply.
>
> So, for point of time/resources/usability there is no much choice.
>
> I think folding the "Erlang repos developed by ASF" list will solve
> most part of the problems. However, I think it worth to keep these
> apps in own repos:
> - rexi
> - b64url
> - config
> - snappy
> - khash
> - ets-lru
> - twig (why we still need it?)
>
> As they could be reused outside and they shouldn't involve any
> dependencies with other couch modules by design. Everyone else may
> stand on where they are.
>
> P.S. I'm not sure if git-subtree will not introduce more new problems
> as it's quite tricky thing to live with it.
>
> --
> ,,,^..^,,,
Re: On dependency management and CI issues associated with it
Posted by Alexander Shorin <kx...@gmail.com>.
Hi Paul!
Thanks for great input!
On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis <pa...@gmail.com> wrote:
> If anyone has a strong objection to a monolithic Erlang repo I'd like
> to hear it. Otherwise I may work up a lengthier and more thorough
> proposal for dev@ to consider consolidating all of these repositories
> for sanity and profit.
It's hard to object against that since this actually solves a lot of
problems solution of which will require more work to do and still will
leave a place for mistakes or require quite specific toolchain to
work.
Making our current repos design work right will require even more work to apply.
So, for point of time/resources/usability there is no much choice.
I think folding the "Erlang repos developed by ASF" list will solve
most part of the problems. However, I think it worth to keep these
apps in own repos:
- rexi
- b64url
- config
- snappy
- khash
- ets-lru
- twig (why we still need it?)
As they could be reused outside and they shouldn't involve any
dependencies with other couch modules by design. Everyone else may
stand on where they are.
P.S. I'm not sure if git-subtree will not introduce more new problems
as it's quite tricky thing to live with it.
--
,,,^..^,,,
Re: On dependency management and CI issues associated with it
Posted by Robert Newson <rn...@apache.org>.
Keeping fauxton in a separate repo makes sense. It has a different release cycle. It's genuinely decoupled. Getting all the Erlang into one repo is really the goal.
With couch_epi as a core application, anyone can extend and customise couchdb by adding another dependency. At most, we might identify new places for epi hooks as we go.
> On 13 Apr 2016, at 17:11, Paul Davis <pa...@gmail.com> wrote:
>
> Hello everybody!
>
> Wow, 56 repos! Hopefully we get an award somewhere for that. I've
> listed the repositories below in some crude groups to try and give an
> idea of what we're working with. I have to agree that this is getting
> a bit on the ridiculous side. Of all of the repos that the ASF
> actually develops I'm only seeing four or so Erlang apps (b64url,
> config, couch-collate, and khash) that are likely truly re-usable
> outside of CouchDB without mucking about.
>
> I have previously (years ago) played around with trying to go back to
> a single repository. It generally works fine, the only issue that I
> found was that rebar's {vsn, git} tag for *.app.src files doesn't work
> in a single repo and it gets a bit complicated managing that by hand.
> However, I think it would be possible to add something like rebar's
> deps files and a small custom tool that either a) breaks the build if
> versions haven't changed or b) even better, automatically sets
> application versions based on a source file (and tweaks them like git
> describe when there's been a commit since the version). This lets us
> continue to "alias" commits with a human readable version and doesn't
> require a single version across all applications. (Alternatively, we
> could wipe the version info on every project and set a single version
> that is the same for all applications that matches the CouchDB
> version, but this might get weird for upstream dependencies).
>
> That said, I'd agree with Bob that the new dependency format seems to
> be solving a problem we shouldn't have. I'd rather just pull
> everything into a single repo and use tooling to help maintain any
> sharp edges like the versioning issue I mentioned above.
>
> Personally, what I'd like to see is to have all Erlang repos merged
> into the main couchdb.git repo and then have all upstream dependencies
> managed by git-subtree. I could go either way on having the node and
> spidermonkey view engines included or not. For the non Erlang parts of
> our release (fauxton and documentation) I'd keep them as separate
> repos so that their tooling doesn't need to be changed and/or adapted
> to work out of a repo subdirectory. The administration things also
> seem to make good sense to keep separate as they're not part of the
> product/release tarball/whatever.
>
> If anyone has a strong objection to a monolithic Erlang repo I'd like
> to hear it. Otherwise I may work up a lengthier and more thorough
> proposal for dev@ to consider consolidating all of these repositories
> for sanity and profit.
>
> Paul
>
> Main repo:
>
> couchdb.git
>
>
> Erlang repos developed by ASF:
>
> couchdb-b64url.git
> couchdb-cassim.git
> couchdb-chttpd.git
> couchdb-config.git
> couchdb-couch-collate.git
> couchdb-couch-dbupdates.git
> couchdb-couch-epi.git
> couchdb-couch-event.git
> couchdb-couch-httpd.git
> couchdb-couch-index.git
> couchdb-couch-log-lager.git
> couchdb-couch-log.git
> couchdb-couch-mrview.git
> couchdb-couch-plugins.git
> couchdb-couch-replicator.git
> couchdb-couch-stats.git
> couchdb-couch.git
> couchdb-ddoc-cache.git
> couchdb-erlang-tests.git
> couchdb-ets-lru.git
> couchdb-fabric.git
> couchdb-global-changes.git
> couchdb-ioq.git
> couchdb-khash.git
> couchdb-mango.git
> couchdb-mem3.git
> couchdb-peruser.git
> couchdb-rexi.git
> couchdb-setup.git
> couchdb-snappy.git
> couchdb-twig.git
>
>
> Non-Erlang things we develop as part of a release:
>
> couchdb-fauxton.git
> couchdb-documentation.git
>
>
> Mirrored repos of upstream Erlang deps:
>
> couchdb-bear.git
> couchdb-folsom.git
> couchdb-goldrush.git
> couchdb-ibrowse.git
> couchdb-jiffy.git
> couchdb-lager.git
> couchdb-meck.git
> couchdb-mochiweb.git
> couchdb-oauth.git
> couchdb-rebar.git
>
>
> Query Servers:
>
> couchdb-query-server-node.git
> couchdb-query-server-spidermonkey.git
>
>
> Unsure but has Erlang in it:
>
> couchdb-examples.git
>
>
> Project Administrative Things Kinda:
>
> couchdb-admin.git
> couchdb-ci.git
> couchdb-docker.git
> couchdb-www.git
>
>
> Client Library:
>
> couchdb-nano.git
>
>
> JS CLI tool:
>
> couchdb-nmo.git
>
>
> Empty:
>
> couchdb-javascript-tests.git
>
> Legacy:
>
> couchdb-futon.git
> couchdb-jquery-couch.git
>
>
>> On Wed, Apr 13, 2016 at 3:41 AM, Garren Smith <ga...@apache.org> wrote:
>> I like the idea of going back to a single repo for core db features. I
>> would like Fauxton to still be in its own repo.
>> As someone who wrote some very basic erlang code for CouchDB recently. I
>> found the multiple repos quite tricky to manage and I couldn't see how it
>> made anything easier.
>>
>>> On Wed, Apr 13, 2016 at 8:35 AM, Alexander Shorin <kx...@gmail.com> wrote:
>>>
>>> Hi Robert,
>>>
>>> Point about flattening to a single repository is valid: in the end, we
>>> have our apps repos in broken state all the time as they are not
>>> declare their decencies. So noone can pick fabric@master and run it -
>>> he'll spend quite a lot of time to figure the deps of the right
>>> versions. But the idea to solve the problem by reducing set of
>>> repositories we have to test is good.
>>>
>>>
>>> Hi Iliya,
>>>
>>> I have alternative solution for you:
>>>
>>> - Turn-off Travis CI everywhere where we cannot be sure about testing
>>> without depended PRs (all except third-party modules, fauxton, docs,
>>> and few more independent projects like couch-epi);
>>> - Require everyone to submit additional PR to apache/couchdb repo with
>>> commit hashes update;
>>> - On this apache/couchdb PR we'll run CI testing;
>>> - If you rebase/update any of your subcomponent PRs you must update
>>> commit hash on apache/couchdb one;
>>>
>>> Pros:
>>> - We won't forget to update rebar.config when new changes lands;
>>> - We will always run complete integration testing with all the right
>>> deps states;
>>> - We won't have to invent any complicated integration solutions to
>>> deal with sub-repos testing;
>>> - No new new steps/files/work introduced, so there is no need to care
>>> about learning curve;
>>>
>>> Cons:
>>> - Need to be a bit tricky on Travis builder to realize on which remote
>>> (fork) new rebar.config hashes are to correctly checkout them, though
>>> that is not a rocket science since we have access to git information
>>> there.
>>>
>>> The Jenkins CI role here is to ensure that we have master build right
>>> and releases build right, on the various OSes.
>>>
>>> Sounds simpler and better for me, how it does for you?
>>>
>>> --
>>> ,,,^..^,,,
>>>
>>>
>>> On Wed, Apr 13, 2016 at 12:37 AM, Robert Samuel Newson
>>> <rn...@apache.org> wrote:
>>>> I'd like us to instead consider flattening to a single repository. I've
>>> found no value and only pain from the multiple repositories approach (43 in
>>> total!).
>>>>
>>>> The contention is that multiple repositories enforces application
>>> boundaries (low coupling / high cohesion) but I've not felt that in
>>> reality. We don't, and couldn't meaningfully, release any of our components
>>> separately, and, as Ilya makes clear, many enhancements require changes to
>>> multiple repositories, and we break this into multiple commits, losing the
>>> ability to look at an enhancement in toto.
>>>>
>>>> If what Ilya is proposing is the solution, I think it's the solution to
>>> a problem we should not have.
>>>>
>>>> B.
>>>>
>>>>> On 12 Apr 2016, at 16:22, Ilya Khlopotov <ii...@ca.ibm.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Dear community,
>>>>>
>>>>>
>>>>> There is a problem with contributors workflow which renders our CI
>>> system
>>>>> useless. As you might know couchdb project consists of multiple
>>>>> repositories. Most of the time changes cross the repositories
>>> boundaries.
>>>>> When this happens the push to any of the repositories causes CI
>>> failures.
>>>>> CI fails since it uses the old version of dependencies from main
>>> repository
>>>>> of the project. Here is what we can do about it.
>>>>>
>>>>> # Proposal
>>>>>
>>>>> Let's use multiple files for dependency management.
>>>>>
>>>>> - deps.json - serves the same purpose as dependencies list from current
>>>>> rebar.config.script
>>>>> - proposed.deps.json - here we specify list of PRs we want to commit
>>>>> atomically
>>>>> - override.deps.json - local file outside of version control which we
>>>>> consult in order to include development tools specific to contributor
>>> (code
>>>>> reloader, debugger, tracer, profiler, binpp, ...)
>>>>>
>>>>> Bellow is the example of a content of these files:
>>>>>
>>>>> ## deps.json
>>>>> {
>>>>> "src/b64url": [
>>>>> "https://github.com/apache/couchdb-b64url",
>>>>> "6895652d80f95cdf04efb14625abed868998f174"
>>>>> ],
>>>>> "src/cassim": [
>>>>> "https://github.com/apache/couchdb-cassim",
>>>>> "9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
>>>>> ],
>>>>> "src/chttpd": [
>>>>> "https://github.com/apache/couchdb-chttpd",
>>>>> "54e8f6147486d9afc5245e0143d15a4dd1185654"
>>>>> ],
>>>>> "src/meck": [
>>>>> "https://github.com/apache/couchdb-meck",
>>>>> "tree/0.8.2"
>>>>> ],
>>>>> ....
>>>>> }
>>>>>
>>>>> ## proposed.deps.json
>>>>> {
>>>>> "src/couch": "https://github.com/apache/couchdb-couch/pull/124",
>>>>> "src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"
>>>>> "src/couch_tests": [
>>>>> "https://github.com/apache/couchdb-erlang-tests",
>>>>> "tree/branch"
>>>>> ],
>>>>> }
>>>>>
>>>>> # Interface
>>>>>
>>>>> I propose to write a simple CLI tool to work with this structure.
>>> Bellow is
>>>>> a list of commands which we need to support (for minimal version)
>>>>>
>>>>> ## Adding new dependency
>>>>>
>>>>> git propose add https://github.com/apache/couchdb-foo
>>>>> a2d5ad2eedc960248b806f61df0a1009462bdb46
>>>>> git propose add https://github.com/apache/couchdb-bar tree/branch_name
>>>>>
>>>>> ## Adding new PR to the change set
>>>>>
>>>>> git propose add https://github.com/apache/couchdb-config/pull/4
>>>>>
>>>>> ## Checking out right dependencies
>>>>>
>>>>> git propose checkout
>>>>>
>>>>> ## Checking out release
>>>>>
>>>>> git propose checkout --release # this would ignore proposed.deps.json
>>> if it
>>>>> exists
>>>>>
>>>>> ## Merge the change
>>>>>
>>>>> This command would do the following:
>>>>> - Parse proposed.deps.json
>>>>> - Retrieve merge commit sha for every PR (exit if dependency is not
>>> merged
>>>>> yet)
>>>>> - Update dependencies in deps.json with correct merge commit sha
>>>>> - remove proposed.deps.json
>>>>>
>>>>> # Workflow
>>>>>
>>>>> export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv`
>>>>> git checkout -b feature-ZZZ
>>>>> cd src/X && hack dependency X
>>>>> cd ../..
>>>>> cd src/Y && hack dependency Y
>>>>> issue PRs for X and Y
>>>>> cd ../..
>>>>> git propose add https://github.com/apache/couchdb-X/pull/4
>>>>> git propose add https://github.com/apache/couchdb-Y/pull/49
>>>>> git add proposed.deps.json
>>>>> git commit -m "Commit feature {something} which does {a thing} and can
>>> be
>>>>> tested as {procedure}"
>>>>> git push origin feature-ZZZ
>>>>> ^ this would trigger our CI
>>>>> CI would do
>>>>> git propose checkout && ./configure && make check
>>>>>
>>>>> # Pros and Cons
>>>>>
>>>>> ## Pros
>>>>>
>>>>> - Changes are merged atomically
>>>>> - CI runs against expected versions of deps
>>>>> - Enables git bisect
>>>>> - Reduce tasks that needs to be done by ASF committer (no need to update
>>>>> rebar.config.script manually)
>>>>> - Simplifies testing of PRs by reviewers
>>>>> - Simplifies rebar.config since rebar is not used for managing deps
>>>>>
>>>>> ## Cons
>>>>>
>>>>> - some github.com specifics (concept of PRs and access to github API
>>> to get
>>>>> info about PR)
>>>>> - we need to have github.com as one of the remotes
>>>>> - we trigger CI only on push to main repository
>>>>>
>>>>> # Implementation
>>>>>
>>>>> We write a git-propose script in python and place it in ./bin. We add
>>> ./bin
>>>>> into either GIT_EXEC_PATH or PATH. You always can call the script
>>> directly
>>>>> (as ./bin/git-propose) if you don't like amending
>>>>> your environment.
>>>>>
>>>>> # Later improvements
>>>>>
>>>>> - We can issue PRs from the tool itself
>>>>> - We can merge from the tool itself
>>>>> - We can implement support for multiple remotes (asf, github, private)
>>>>> - We can implement support for multiple git transports (for the first
>>>>> version all repositories in *.deps.json files would use https://)
>>>>>
>>>>> Sincerely,
>>>>> ILYA KHLOPOTOV
>>>
Re: On dependency management and CI issues associated with it
Posted by Paul Davis <pa...@gmail.com>.
Hello everybody!
Wow, 56 repos! Hopefully we get an award somewhere for that. I've
listed the repositories below in some crude groups to try and give an
idea of what we're working with. I have to agree that this is getting
a bit on the ridiculous side. Of all of the repos that the ASF
actually develops I'm only seeing four or so Erlang apps (b64url,
config, couch-collate, and khash) that are likely truly re-usable
outside of CouchDB without mucking about.
I have previously (years ago) played around with trying to go back to
a single repository. It generally works fine, the only issue that I
found was that rebar's {vsn, git} tag for *.app.src files doesn't work
in a single repo and it gets a bit complicated managing that by hand.
However, I think it would be possible to add something like rebar's
deps files and a small custom tool that either a) breaks the build if
versions haven't changed or b) even better, automatically sets
application versions based on a source file (and tweaks them like git
describe when there's been a commit since the version). This lets us
continue to "alias" commits with a human readable version and doesn't
require a single version across all applications. (Alternatively, we
could wipe the version info on every project and set a single version
that is the same for all applications that matches the CouchDB
version, but this might get weird for upstream dependencies).
That said, I'd agree with Bob that the new dependency format seems to
be solving a problem we shouldn't have. I'd rather just pull
everything into a single repo and use tooling to help maintain any
sharp edges like the versioning issue I mentioned above.
Personally, what I'd like to see is to have all Erlang repos merged
into the main couchdb.git repo and then have all upstream dependencies
managed by git-subtree. I could go either way on having the node and
spidermonkey view engines included or not. For the non Erlang parts of
our release (fauxton and documentation) I'd keep them as separate
repos so that their tooling doesn't need to be changed and/or adapted
to work out of a repo subdirectory. The administration things also
seem to make good sense to keep separate as they're not part of the
product/release tarball/whatever.
If anyone has a strong objection to a monolithic Erlang repo I'd like
to hear it. Otherwise I may work up a lengthier and more thorough
proposal for dev@ to consider consolidating all of these repositories
for sanity and profit.
Paul
Main repo:
couchdb.git
Erlang repos developed by ASF:
couchdb-b64url.git
couchdb-cassim.git
couchdb-chttpd.git
couchdb-config.git
couchdb-couch-collate.git
couchdb-couch-dbupdates.git
couchdb-couch-epi.git
couchdb-couch-event.git
couchdb-couch-httpd.git
couchdb-couch-index.git
couchdb-couch-log-lager.git
couchdb-couch-log.git
couchdb-couch-mrview.git
couchdb-couch-plugins.git
couchdb-couch-replicator.git
couchdb-couch-stats.git
couchdb-couch.git
couchdb-ddoc-cache.git
couchdb-erlang-tests.git
couchdb-ets-lru.git
couchdb-fabric.git
couchdb-global-changes.git
couchdb-ioq.git
couchdb-khash.git
couchdb-mango.git
couchdb-mem3.git
couchdb-peruser.git
couchdb-rexi.git
couchdb-setup.git
couchdb-snappy.git
couchdb-twig.git
Non-Erlang things we develop as part of a release:
couchdb-fauxton.git
couchdb-documentation.git
Mirrored repos of upstream Erlang deps:
couchdb-bear.git
couchdb-folsom.git
couchdb-goldrush.git
couchdb-ibrowse.git
couchdb-jiffy.git
couchdb-lager.git
couchdb-meck.git
couchdb-mochiweb.git
couchdb-oauth.git
couchdb-rebar.git
Query Servers:
couchdb-query-server-node.git
couchdb-query-server-spidermonkey.git
Unsure but has Erlang in it:
couchdb-examples.git
Project Administrative Things Kinda:
couchdb-admin.git
couchdb-ci.git
couchdb-docker.git
couchdb-www.git
Client Library:
couchdb-nano.git
JS CLI tool:
couchdb-nmo.git
Empty:
couchdb-javascript-tests.git
Legacy:
couchdb-futon.git
couchdb-jquery-couch.git
On Wed, Apr 13, 2016 at 3:41 AM, Garren Smith <ga...@apache.org> wrote:
> I like the idea of going back to a single repo for core db features. I
> would like Fauxton to still be in its own repo.
> As someone who wrote some very basic erlang code for CouchDB recently. I
> found the multiple repos quite tricky to manage and I couldn't see how it
> made anything easier.
>
> On Wed, Apr 13, 2016 at 8:35 AM, Alexander Shorin <kx...@gmail.com> wrote:
>
>> Hi Robert,
>>
>> Point about flattening to a single repository is valid: in the end, we
>> have our apps repos in broken state all the time as they are not
>> declare their decencies. So noone can pick fabric@master and run it -
>> he'll spend quite a lot of time to figure the deps of the right
>> versions. But the idea to solve the problem by reducing set of
>> repositories we have to test is good.
>>
>>
>> Hi Iliya,
>>
>> I have alternative solution for you:
>>
>> - Turn-off Travis CI everywhere where we cannot be sure about testing
>> without depended PRs (all except third-party modules, fauxton, docs,
>> and few more independent projects like couch-epi);
>> - Require everyone to submit additional PR to apache/couchdb repo with
>> commit hashes update;
>> - On this apache/couchdb PR we'll run CI testing;
>> - If you rebase/update any of your subcomponent PRs you must update
>> commit hash on apache/couchdb one;
>>
>> Pros:
>> - We won't forget to update rebar.config when new changes lands;
>> - We will always run complete integration testing with all the right
>> deps states;
>> - We won't have to invent any complicated integration solutions to
>> deal with sub-repos testing;
>> - No new new steps/files/work introduced, so there is no need to care
>> about learning curve;
>>
>> Cons:
>> - Need to be a bit tricky on Travis builder to realize on which remote
>> (fork) new rebar.config hashes are to correctly checkout them, though
>> that is not a rocket science since we have access to git information
>> there.
>>
>> The Jenkins CI role here is to ensure that we have master build right
>> and releases build right, on the various OSes.
>>
>> Sounds simpler and better for me, how it does for you?
>>
>> --
>> ,,,^..^,,,
>>
>>
>> On Wed, Apr 13, 2016 at 12:37 AM, Robert Samuel Newson
>> <rn...@apache.org> wrote:
>> > I'd like us to instead consider flattening to a single repository. I've
>> found no value and only pain from the multiple repositories approach (43 in
>> total!).
>> >
>> > The contention is that multiple repositories enforces application
>> boundaries (low coupling / high cohesion) but I've not felt that in
>> reality. We don't, and couldn't meaningfully, release any of our components
>> separately, and, as Ilya makes clear, many enhancements require changes to
>> multiple repositories, and we break this into multiple commits, losing the
>> ability to look at an enhancement in toto.
>> >
>> > If what Ilya is proposing is the solution, I think it's the solution to
>> a problem we should not have.
>> >
>> > B.
>> >
>> >> On 12 Apr 2016, at 16:22, Ilya Khlopotov <ii...@ca.ibm.com> wrote:
>> >>
>> >>
>> >>
>> >> Dear community,
>> >>
>> >>
>> >> There is a problem with contributors workflow which renders our CI
>> system
>> >> useless. As you might know couchdb project consists of multiple
>> >> repositories. Most of the time changes cross the repositories
>> boundaries.
>> >> When this happens the push to any of the repositories causes CI
>> failures.
>> >> CI fails since it uses the old version of dependencies from main
>> repository
>> >> of the project. Here is what we can do about it.
>> >>
>> >> # Proposal
>> >>
>> >> Let's use multiple files for dependency management.
>> >>
>> >> - deps.json - serves the same purpose as dependencies list from current
>> >> rebar.config.script
>> >> - proposed.deps.json - here we specify list of PRs we want to commit
>> >> atomically
>> >> - override.deps.json - local file outside of version control which we
>> >> consult in order to include development tools specific to contributor
>> (code
>> >> reloader, debugger, tracer, profiler, binpp, ...)
>> >>
>> >> Bellow is the example of a content of these files:
>> >>
>> >> ## deps.json
>> >> {
>> >> "src/b64url": [
>> >> "https://github.com/apache/couchdb-b64url",
>> >> "6895652d80f95cdf04efb14625abed868998f174"
>> >> ],
>> >> "src/cassim": [
>> >> "https://github.com/apache/couchdb-cassim",
>> >> "9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
>> >> ],
>> >> "src/chttpd": [
>> >> "https://github.com/apache/couchdb-chttpd",
>> >> "54e8f6147486d9afc5245e0143d15a4dd1185654"
>> >> ],
>> >> "src/meck": [
>> >> "https://github.com/apache/couchdb-meck",
>> >> "tree/0.8.2"
>> >> ],
>> >> ....
>> >> }
>> >>
>> >> ## proposed.deps.json
>> >> {
>> >> "src/couch": "https://github.com/apache/couchdb-couch/pull/124",
>> >> "src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"
>> >> "src/couch_tests": [
>> >> "https://github.com/apache/couchdb-erlang-tests",
>> >> "tree/branch"
>> >> ],
>> >> }
>> >>
>> >> # Interface
>> >>
>> >> I propose to write a simple CLI tool to work with this structure.
>> Bellow is
>> >> a list of commands which we need to support (for minimal version)
>> >>
>> >> ## Adding new dependency
>> >>
>> >> git propose add https://github.com/apache/couchdb-foo
>> >> a2d5ad2eedc960248b806f61df0a1009462bdb46
>> >> git propose add https://github.com/apache/couchdb-bar tree/branch_name
>> >>
>> >> ## Adding new PR to the change set
>> >>
>> >> git propose add https://github.com/apache/couchdb-config/pull/4
>> >>
>> >> ## Checking out right dependencies
>> >>
>> >> git propose checkout
>> >>
>> >> ## Checking out release
>> >>
>> >> git propose checkout --release # this would ignore proposed.deps.json
>> if it
>> >> exists
>> >>
>> >> ## Merge the change
>> >>
>> >> This command would do the following:
>> >> - Parse proposed.deps.json
>> >> - Retrieve merge commit sha for every PR (exit if dependency is not
>> merged
>> >> yet)
>> >> - Update dependencies in deps.json with correct merge commit sha
>> >> - remove proposed.deps.json
>> >>
>> >> # Workflow
>> >>
>> >> export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv`
>> >> git checkout -b feature-ZZZ
>> >> cd src/X && hack dependency X
>> >> cd ../..
>> >> cd src/Y && hack dependency Y
>> >> issue PRs for X and Y
>> >> cd ../..
>> >> git propose add https://github.com/apache/couchdb-X/pull/4
>> >> git propose add https://github.com/apache/couchdb-Y/pull/49
>> >> git add proposed.deps.json
>> >> git commit -m "Commit feature {something} which does {a thing} and can
>> be
>> >> tested as {procedure}"
>> >> git push origin feature-ZZZ
>> >> ^ this would trigger our CI
>> >> CI would do
>> >> git propose checkout && ./configure && make check
>> >>
>> >> # Pros and Cons
>> >>
>> >> ## Pros
>> >>
>> >> - Changes are merged atomically
>> >> - CI runs against expected versions of deps
>> >> - Enables git bisect
>> >> - Reduce tasks that needs to be done by ASF committer (no need to update
>> >> rebar.config.script manually)
>> >> - Simplifies testing of PRs by reviewers
>> >> - Simplifies rebar.config since rebar is not used for managing deps
>> >>
>> >> ## Cons
>> >>
>> >> - some github.com specifics (concept of PRs and access to github API
>> to get
>> >> info about PR)
>> >> - we need to have github.com as one of the remotes
>> >> - we trigger CI only on push to main repository
>> >>
>> >> # Implementation
>> >>
>> >> We write a git-propose script in python and place it in ./bin. We add
>> ./bin
>> >> into either GIT_EXEC_PATH or PATH. You always can call the script
>> directly
>> >> (as ./bin/git-propose) if you don't like amending
>> >> your environment.
>> >>
>> >> # Later improvements
>> >>
>> >> - We can issue PRs from the tool itself
>> >> - We can merge from the tool itself
>> >> - We can implement support for multiple remotes (asf, github, private)
>> >> - We can implement support for multiple git transports (for the first
>> >> version all repositories in *.deps.json files would use https://)
>> >>
>> >> Sincerely,
>> >> ILYA KHLOPOTOV
>> >
>>
Re: On dependency management and CI issues associated with it
Posted by Garren Smith <ga...@apache.org>.
I like the idea of going back to a single repo for core db features. I
would like Fauxton to still be in its own repo.
As someone who wrote some very basic erlang code for CouchDB recently. I
found the multiple repos quite tricky to manage and I couldn't see how it
made anything easier.
On Wed, Apr 13, 2016 at 8:35 AM, Alexander Shorin <kx...@gmail.com> wrote:
> Hi Robert,
>
> Point about flattening to a single repository is valid: in the end, we
> have our apps repos in broken state all the time as they are not
> declare their decencies. So noone can pick fabric@master and run it -
> he'll spend quite a lot of time to figure the deps of the right
> versions. But the idea to solve the problem by reducing set of
> repositories we have to test is good.
>
>
> Hi Iliya,
>
> I have alternative solution for you:
>
> - Turn-off Travis CI everywhere where we cannot be sure about testing
> without depended PRs (all except third-party modules, fauxton, docs,
> and few more independent projects like couch-epi);
> - Require everyone to submit additional PR to apache/couchdb repo with
> commit hashes update;
> - On this apache/couchdb PR we'll run CI testing;
> - If you rebase/update any of your subcomponent PRs you must update
> commit hash on apache/couchdb one;
>
> Pros:
> - We won't forget to update rebar.config when new changes lands;
> - We will always run complete integration testing with all the right
> deps states;
> - We won't have to invent any complicated integration solutions to
> deal with sub-repos testing;
> - No new new steps/files/work introduced, so there is no need to care
> about learning curve;
>
> Cons:
> - Need to be a bit tricky on Travis builder to realize on which remote
> (fork) new rebar.config hashes are to correctly checkout them, though
> that is not a rocket science since we have access to git information
> there.
>
> The Jenkins CI role here is to ensure that we have master build right
> and releases build right, on the various OSes.
>
> Sounds simpler and better for me, how it does for you?
>
> --
> ,,,^..^,,,
>
>
> On Wed, Apr 13, 2016 at 12:37 AM, Robert Samuel Newson
> <rn...@apache.org> wrote:
> > I'd like us to instead consider flattening to a single repository. I've
> found no value and only pain from the multiple repositories approach (43 in
> total!).
> >
> > The contention is that multiple repositories enforces application
> boundaries (low coupling / high cohesion) but I've not felt that in
> reality. We don't, and couldn't meaningfully, release any of our components
> separately, and, as Ilya makes clear, many enhancements require changes to
> multiple repositories, and we break this into multiple commits, losing the
> ability to look at an enhancement in toto.
> >
> > If what Ilya is proposing is the solution, I think it's the solution to
> a problem we should not have.
> >
> > B.
> >
> >> On 12 Apr 2016, at 16:22, Ilya Khlopotov <ii...@ca.ibm.com> wrote:
> >>
> >>
> >>
> >> Dear community,
> >>
> >>
> >> There is a problem with contributors workflow which renders our CI
> system
> >> useless. As you might know couchdb project consists of multiple
> >> repositories. Most of the time changes cross the repositories
> boundaries.
> >> When this happens the push to any of the repositories causes CI
> failures.
> >> CI fails since it uses the old version of dependencies from main
> repository
> >> of the project. Here is what we can do about it.
> >>
> >> # Proposal
> >>
> >> Let's use multiple files for dependency management.
> >>
> >> - deps.json - serves the same purpose as dependencies list from current
> >> rebar.config.script
> >> - proposed.deps.json - here we specify list of PRs we want to commit
> >> atomically
> >> - override.deps.json - local file outside of version control which we
> >> consult in order to include development tools specific to contributor
> (code
> >> reloader, debugger, tracer, profiler, binpp, ...)
> >>
> >> Bellow is the example of a content of these files:
> >>
> >> ## deps.json
> >> {
> >> "src/b64url": [
> >> "https://github.com/apache/couchdb-b64url",
> >> "6895652d80f95cdf04efb14625abed868998f174"
> >> ],
> >> "src/cassim": [
> >> "https://github.com/apache/couchdb-cassim",
> >> "9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
> >> ],
> >> "src/chttpd": [
> >> "https://github.com/apache/couchdb-chttpd",
> >> "54e8f6147486d9afc5245e0143d15a4dd1185654"
> >> ],
> >> "src/meck": [
> >> "https://github.com/apache/couchdb-meck",
> >> "tree/0.8.2"
> >> ],
> >> ....
> >> }
> >>
> >> ## proposed.deps.json
> >> {
> >> "src/couch": "https://github.com/apache/couchdb-couch/pull/124",
> >> "src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"
> >> "src/couch_tests": [
> >> "https://github.com/apache/couchdb-erlang-tests",
> >> "tree/branch"
> >> ],
> >> }
> >>
> >> # Interface
> >>
> >> I propose to write a simple CLI tool to work with this structure.
> Bellow is
> >> a list of commands which we need to support (for minimal version)
> >>
> >> ## Adding new dependency
> >>
> >> git propose add https://github.com/apache/couchdb-foo
> >> a2d5ad2eedc960248b806f61df0a1009462bdb46
> >> git propose add https://github.com/apache/couchdb-bar tree/branch_name
> >>
> >> ## Adding new PR to the change set
> >>
> >> git propose add https://github.com/apache/couchdb-config/pull/4
> >>
> >> ## Checking out right dependencies
> >>
> >> git propose checkout
> >>
> >> ## Checking out release
> >>
> >> git propose checkout --release # this would ignore proposed.deps.json
> if it
> >> exists
> >>
> >> ## Merge the change
> >>
> >> This command would do the following:
> >> - Parse proposed.deps.json
> >> - Retrieve merge commit sha for every PR (exit if dependency is not
> merged
> >> yet)
> >> - Update dependencies in deps.json with correct merge commit sha
> >> - remove proposed.deps.json
> >>
> >> # Workflow
> >>
> >> export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv`
> >> git checkout -b feature-ZZZ
> >> cd src/X && hack dependency X
> >> cd ../..
> >> cd src/Y && hack dependency Y
> >> issue PRs for X and Y
> >> cd ../..
> >> git propose add https://github.com/apache/couchdb-X/pull/4
> >> git propose add https://github.com/apache/couchdb-Y/pull/49
> >> git add proposed.deps.json
> >> git commit -m "Commit feature {something} which does {a thing} and can
> be
> >> tested as {procedure}"
> >> git push origin feature-ZZZ
> >> ^ this would trigger our CI
> >> CI would do
> >> git propose checkout && ./configure && make check
> >>
> >> # Pros and Cons
> >>
> >> ## Pros
> >>
> >> - Changes are merged atomically
> >> - CI runs against expected versions of deps
> >> - Enables git bisect
> >> - Reduce tasks that needs to be done by ASF committer (no need to update
> >> rebar.config.script manually)
> >> - Simplifies testing of PRs by reviewers
> >> - Simplifies rebar.config since rebar is not used for managing deps
> >>
> >> ## Cons
> >>
> >> - some github.com specifics (concept of PRs and access to github API
> to get
> >> info about PR)
> >> - we need to have github.com as one of the remotes
> >> - we trigger CI only on push to main repository
> >>
> >> # Implementation
> >>
> >> We write a git-propose script in python and place it in ./bin. We add
> ./bin
> >> into either GIT_EXEC_PATH or PATH. You always can call the script
> directly
> >> (as ./bin/git-propose) if you don't like amending
> >> your environment.
> >>
> >> # Later improvements
> >>
> >> - We can issue PRs from the tool itself
> >> - We can merge from the tool itself
> >> - We can implement support for multiple remotes (asf, github, private)
> >> - We can implement support for multiple git transports (for the first
> >> version all repositories in *.deps.json files would use https://)
> >>
> >> Sincerely,
> >> ILYA KHLOPOTOV
> >
>
Re: On dependency management and CI issues associated with it
Posted by Alexander Shorin <kx...@gmail.com>.
On Wed, Apr 13, 2016 at 6:46 PM, Ilya Khlopotov <ii...@ca.ibm.com> wrote:
>
> > - If you rebase/update any of your subcomponent PRs you must update
> commit hash on apache/couchdb one;
>
> Above is an error prune step which actually makes the following false
> > - No new new steps/files/work introduced, so there is no need to care
> about learning curve;
>
> > Sounds simpler and better for me, how it does for you?
> Simpler - maybe. Better not so much.
Yes, I also found a case, when you have at least twice trigger CI
build: once for initial PR, and second after all related PRs get
merged, because merge operation produces merge commit which you should
write into rebar.config instead.
I think this could be automated as well (wonder if repo tool can be
extended for that kind of task), but there is still need to think
about and do some research.
> We can store deps in proposed format and make rebar understand json (rebar3 understands json BTW) but I am sure this approach would be at least frowned upon if not rejected.
If rebar3 already supports proposed by you format, then good. More
points to upgrade that tortoise. But agree on with your latter part.
> Also I would really love to merge in my local deps I use in development. But this is a different topic probably.
Good point about deps overlaying. It's nice feature to have.
P.S. Seems like the most simple way is to fold all the repos back to
one, but I won't think about the hell someone have to pass though to
set commit history in the right order without breaking anything...
--
,,,^..^,,,
Re: On dependency management and CI issues associated with it
Posted by Ilya Khlopotov <ii...@ca.ibm.com>.
Hi Alex,
Thank you for your feedback.
> - If you rebase/update any of your subcomponent PRs you must update
commit hash on apache/couchdb one;
Above is an error prune step which actually makes the following false
> - No new new steps/files/work introduced, so there is no need to care
about learning curve;
> Sounds simpler and better for me, how it does for you?
Simpler - maybe. Better not so much. I would like to add some other
automatons on top latter on. Such as automatic generation of a Release
Notes between two tags for example. In order to do that I would need a list
of dependencies in simple format.
I always can call `bin/rebar -q -C rebar.config.script list-deps` from my
python scripts but I cannot do it every time I need to lookup something
since it slows things down significantly. So I have to read once and cache
it in memory and do other trickery. Relying on trickery makes the script
complicated which would raise additional questions on the PR with it.
We can store deps in proposed format and make rebar understand json (rebar3
understands json BTW) but I am sure this approach would be at least frowned
upon if not rejected.
Also I would really love to merge in my local deps I use in development.
But this is a different topic probably.
Best regards,
ILYA
From: Alexander Shorin <kx...@gmail.com>
To: "dev@couchdb.apache.org" <de...@couchdb.apache.org>
Date: 2016/04/12 11:35 PM
Subject: Re: On dependency management and CI issues associated with it
Hi Robert,
Point about flattening to a single repository is valid: in the end, we
have our apps repos in broken state all the time as they are not
declare their decencies. So noone can pick fabric@master and run it -
he'll spend quite a lot of time to figure the deps of the right
versions. But the idea to solve the problem by reducing set of
repositories we have to test is good.
Hi Iliya,
I have alternative solution for you:
- Turn-off Travis CI everywhere where we cannot be sure about testing
without depended PRs (all except third-party modules, fauxton, docs,
and few more independent projects like couch-epi);
- Require everyone to submit additional PR to apache/couchdb repo with
commit hashes update;
- On this apache/couchdb PR we'll run CI testing;
- If you rebase/update any of your subcomponent PRs you must update
commit hash on apache/couchdb one;
Pros:
- We won't forget to update rebar.config when new changes lands;
- We will always run complete integration testing with all the right
deps states;
- We won't have to invent any complicated integration solutions to
deal with sub-repos testing;
- No new new steps/files/work introduced, so there is no need to care
about learning curve;
Cons:
- Need to be a bit tricky on Travis builder to realize on which remote
(fork) new rebar.config hashes are to correctly checkout them, though
that is not a rocket science since we have access to git information
there.
The Jenkins CI role here is to ensure that we have master build right
and releases build right, on the various OSes.
Sounds simpler and better for me, how it does for you?
--
,,,^..^,,,
On Wed, Apr 13, 2016 at 12:37 AM, Robert Samuel Newson
<rn...@apache.org> wrote:
> I'd like us to instead consider flattening to a single repository. I've
found no value and only pain from the multiple repositories approach (43 in
total!).
>
> The contention is that multiple repositories enforces application
boundaries (low coupling / high cohesion) but I've not felt that in
reality. We don't, and couldn't meaningfully, release any of our components
separately, and, as Ilya makes clear, many enhancements require changes to
multiple repositories, and we break this into multiple commits, losing the
ability to look at an enhancement in toto.
>
> If what Ilya is proposing is the solution, I think it's the solution to a
problem we should not have.
>
> B.
>
>> On 12 Apr 2016, at 16:22, Ilya Khlopotov <ii...@ca.ibm.com> wrote:
>>
>>
>>
>> Dear community,
>>
>>
>> There is a problem with contributors workflow which renders our CI
system
>> useless. As you might know couchdb project consists of multiple
>> repositories. Most of the time changes cross the repositories
boundaries.
>> When this happens the push to any of the repositories causes CI
failures.
>> CI fails since it uses the old version of dependencies from main
repository
>> of the project. Here is what we can do about it.
>>
>> # Proposal
>>
>> Let's use multiple files for dependency management.
>>
>> - deps.json - serves the same purpose as dependencies list from current
>> rebar.config.script
>> - proposed.deps.json - here we specify list of PRs we want to commit
>> atomically
>> - override.deps.json - local file outside of version control which we
>> consult in order to include development tools specific to contributor
(code
>> reloader, debugger, tracer, profiler, binpp, ...)
>>
>> Bellow is the example of a content of these files:
>>
>> ## deps.json
>> {
>> "src/b64url": [
>> "https://github.com/apache/couchdb-b64url",
>> "6895652d80f95cdf04efb14625abed868998f174"
>> ],
>> "src/cassim": [
>> "https://github.com/apache/couchdb-cassim",
>> "9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
>> ],
>> "src/chttpd": [
>> "https://github.com/apache/couchdb-chttpd",
>> "54e8f6147486d9afc5245e0143d15a4dd1185654"
>> ],
>> "src/meck": [
>> "https://github.com/apache/couchdb-meck",
>> "tree/0.8.2"
>> ],
>> ....
>> }
>>
>> ## proposed.deps.json
>> {
>> "src/couch": "https://github.com/apache/couchdb-couch/pull/124",
>> "src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"
>> "src/couch_tests": [
>> "https://github.com/apache/couchdb-erlang-tests",
>> "tree/branch"
>> ],
>> }
>>
>> # Interface
>>
>> I propose to write a simple CLI tool to work with this structure. Bellow
is
>> a list of commands which we need to support (for minimal version)
>>
>> ## Adding new dependency
>>
>> git propose add https://github.com/apache/couchdb-foo
>> a2d5ad2eedc960248b806f61df0a1009462bdb46
>> git propose add https://github.com/apache/couchdb-bar tree/branch_name
>>
>> ## Adding new PR to the change set
>>
>> git propose add https://github.com/apache/couchdb-config/pull/4
>>
>> ## Checking out right dependencies
>>
>> git propose checkout
>>
>> ## Checking out release
>>
>> git propose checkout --release # this would ignore proposed.deps.json if
it
>> exists
>>
>> ## Merge the change
>>
>> This command would do the following:
>> - Parse proposed.deps.json
>> - Retrieve merge commit sha for every PR (exit if dependency is not
merged
>> yet)
>> - Update dependencies in deps.json with correct merge commit sha
>> - remove proposed.deps.json
>>
>> # Workflow
>>
>> export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv`
>> git checkout -b feature-ZZZ
>> cd src/X && hack dependency X
>> cd ../..
>> cd src/Y && hack dependency Y
>> issue PRs for X and Y
>> cd ../..
>> git propose add https://github.com/apache/couchdb-X/pull/4
>> git propose add https://github.com/apache/couchdb-Y/pull/49
>> git add proposed.deps.json
>> git commit -m "Commit feature {something} which does {a thing} and can
be
>> tested as {procedure}"
>> git push origin feature-ZZZ
>> ^ this would trigger our CI
>> CI would do
>> git propose checkout && ./configure && make check
>>
>> # Pros and Cons
>>
>> ## Pros
>>
>> - Changes are merged atomically
>> - CI runs against expected versions of deps
>> - Enables git bisect
>> - Reduce tasks that needs to be done by ASF committer (no need to update
>> rebar.config.script manually)
>> - Simplifies testing of PRs by reviewers
>> - Simplifies rebar.config since rebar is not used for managing deps
>>
>> ## Cons
>>
>> - some github.com specifics (concept of PRs and access to github API to
get
>> info about PR)
>> - we need to have github.com as one of the remotes
>> - we trigger CI only on push to main repository
>>
>> # Implementation
>>
>> We write a git-propose script in python and place it in ./bin. We
add ./bin
>> into either GIT_EXEC_PATH or PATH. You always can call the script
directly
>> (as ./bin/git-propose) if you don't like amending
>> your environment.
>>
>> # Later improvements
>>
>> - We can issue PRs from the tool itself
>> - We can merge from the tool itself
>> - We can implement support for multiple remotes (asf, github, private)
>> - We can implement support for multiple git transports (for the first
>> version all repositories in *.deps.json files would use https://)
>>
>> Sincerely,
>> ILYA KHLOPOTOV
>
Re: On dependency management and CI issues associated with it
Posted by Alexander Shorin <kx...@gmail.com>.
Hi Robert,
Point about flattening to a single repository is valid: in the end, we
have our apps repos in broken state all the time as they are not
declare their decencies. So noone can pick fabric@master and run it -
he'll spend quite a lot of time to figure the deps of the right
versions. But the idea to solve the problem by reducing set of
repositories we have to test is good.
Hi Iliya,
I have alternative solution for you:
- Turn-off Travis CI everywhere where we cannot be sure about testing
without depended PRs (all except third-party modules, fauxton, docs,
and few more independent projects like couch-epi);
- Require everyone to submit additional PR to apache/couchdb repo with
commit hashes update;
- On this apache/couchdb PR we'll run CI testing;
- If you rebase/update any of your subcomponent PRs you must update
commit hash on apache/couchdb one;
Pros:
- We won't forget to update rebar.config when new changes lands;
- We will always run complete integration testing with all the right
deps states;
- We won't have to invent any complicated integration solutions to
deal with sub-repos testing;
- No new new steps/files/work introduced, so there is no need to care
about learning curve;
Cons:
- Need to be a bit tricky on Travis builder to realize on which remote
(fork) new rebar.config hashes are to correctly checkout them, though
that is not a rocket science since we have access to git information
there.
The Jenkins CI role here is to ensure that we have master build right
and releases build right, on the various OSes.
Sounds simpler and better for me, how it does for you?
--
,,,^..^,,,
On Wed, Apr 13, 2016 at 12:37 AM, Robert Samuel Newson
<rn...@apache.org> wrote:
> I'd like us to instead consider flattening to a single repository. I've found no value and only pain from the multiple repositories approach (43 in total!).
>
> The contention is that multiple repositories enforces application boundaries (low coupling / high cohesion) but I've not felt that in reality. We don't, and couldn't meaningfully, release any of our components separately, and, as Ilya makes clear, many enhancements require changes to multiple repositories, and we break this into multiple commits, losing the ability to look at an enhancement in toto.
>
> If what Ilya is proposing is the solution, I think it's the solution to a problem we should not have.
>
> B.
>
>> On 12 Apr 2016, at 16:22, Ilya Khlopotov <ii...@ca.ibm.com> wrote:
>>
>>
>>
>> Dear community,
>>
>>
>> There is a problem with contributors workflow which renders our CI system
>> useless. As you might know couchdb project consists of multiple
>> repositories. Most of the time changes cross the repositories boundaries.
>> When this happens the push to any of the repositories causes CI failures.
>> CI fails since it uses the old version of dependencies from main repository
>> of the project. Here is what we can do about it.
>>
>> # Proposal
>>
>> Let's use multiple files for dependency management.
>>
>> - deps.json - serves the same purpose as dependencies list from current
>> rebar.config.script
>> - proposed.deps.json - here we specify list of PRs we want to commit
>> atomically
>> - override.deps.json - local file outside of version control which we
>> consult in order to include development tools specific to contributor (code
>> reloader, debugger, tracer, profiler, binpp, ...)
>>
>> Bellow is the example of a content of these files:
>>
>> ## deps.json
>> {
>> "src/b64url": [
>> "https://github.com/apache/couchdb-b64url",
>> "6895652d80f95cdf04efb14625abed868998f174"
>> ],
>> "src/cassim": [
>> "https://github.com/apache/couchdb-cassim",
>> "9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
>> ],
>> "src/chttpd": [
>> "https://github.com/apache/couchdb-chttpd",
>> "54e8f6147486d9afc5245e0143d15a4dd1185654"
>> ],
>> "src/meck": [
>> "https://github.com/apache/couchdb-meck",
>> "tree/0.8.2"
>> ],
>> ....
>> }
>>
>> ## proposed.deps.json
>> {
>> "src/couch": "https://github.com/apache/couchdb-couch/pull/124",
>> "src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"
>> "src/couch_tests": [
>> "https://github.com/apache/couchdb-erlang-tests",
>> "tree/branch"
>> ],
>> }
>>
>> # Interface
>>
>> I propose to write a simple CLI tool to work with this structure. Bellow is
>> a list of commands which we need to support (for minimal version)
>>
>> ## Adding new dependency
>>
>> git propose add https://github.com/apache/couchdb-foo
>> a2d5ad2eedc960248b806f61df0a1009462bdb46
>> git propose add https://github.com/apache/couchdb-bar tree/branch_name
>>
>> ## Adding new PR to the change set
>>
>> git propose add https://github.com/apache/couchdb-config/pull/4
>>
>> ## Checking out right dependencies
>>
>> git propose checkout
>>
>> ## Checking out release
>>
>> git propose checkout --release # this would ignore proposed.deps.json if it
>> exists
>>
>> ## Merge the change
>>
>> This command would do the following:
>> - Parse proposed.deps.json
>> - Retrieve merge commit sha for every PR (exit if dependency is not merged
>> yet)
>> - Update dependencies in deps.json with correct merge commit sha
>> - remove proposed.deps.json
>>
>> # Workflow
>>
>> export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv`
>> git checkout -b feature-ZZZ
>> cd src/X && hack dependency X
>> cd ../..
>> cd src/Y && hack dependency Y
>> issue PRs for X and Y
>> cd ../..
>> git propose add https://github.com/apache/couchdb-X/pull/4
>> git propose add https://github.com/apache/couchdb-Y/pull/49
>> git add proposed.deps.json
>> git commit -m "Commit feature {something} which does {a thing} and can be
>> tested as {procedure}"
>> git push origin feature-ZZZ
>> ^ this would trigger our CI
>> CI would do
>> git propose checkout && ./configure && make check
>>
>> # Pros and Cons
>>
>> ## Pros
>>
>> - Changes are merged atomically
>> - CI runs against expected versions of deps
>> - Enables git bisect
>> - Reduce tasks that needs to be done by ASF committer (no need to update
>> rebar.config.script manually)
>> - Simplifies testing of PRs by reviewers
>> - Simplifies rebar.config since rebar is not used for managing deps
>>
>> ## Cons
>>
>> - some github.com specifics (concept of PRs and access to github API to get
>> info about PR)
>> - we need to have github.com as one of the remotes
>> - we trigger CI only on push to main repository
>>
>> # Implementation
>>
>> We write a git-propose script in python and place it in ./bin. We add ./bin
>> into either GIT_EXEC_PATH or PATH. You always can call the script directly
>> (as ./bin/git-propose) if you don't like amending
>> your environment.
>>
>> # Later improvements
>>
>> - We can issue PRs from the tool itself
>> - We can merge from the tool itself
>> - We can implement support for multiple remotes (asf, github, private)
>> - We can implement support for multiple git transports (for the first
>> version all repositories in *.deps.json files would use https://)
>>
>> Sincerely,
>> ILYA KHLOPOTOV
>
Re: On dependency management and CI issues associated with it
Posted by Robert Samuel Newson <rn...@apache.org>.
I'd like us to instead consider flattening to a single repository. I've found no value and only pain from the multiple repositories approach (43 in total!).
The contention is that multiple repositories enforces application boundaries (low coupling / high cohesion) but I've not felt that in reality. We don't, and couldn't meaningfully, release any of our components separately, and, as Ilya makes clear, many enhancements require changes to multiple repositories, and we break this into multiple commits, losing the ability to look at an enhancement in toto.
If what Ilya is proposing is the solution, I think it's the solution to a problem we should not have.
B.
> On 12 Apr 2016, at 16:22, Ilya Khlopotov <ii...@ca.ibm.com> wrote:
>
>
>
> Dear community,
>
>
> There is a problem with contributors workflow which renders our CI system
> useless. As you might know couchdb project consists of multiple
> repositories. Most of the time changes cross the repositories boundaries.
> When this happens the push to any of the repositories causes CI failures.
> CI fails since it uses the old version of dependencies from main repository
> of the project. Here is what we can do about it.
>
> # Proposal
>
> Let's use multiple files for dependency management.
>
> - deps.json - serves the same purpose as dependencies list from current
> rebar.config.script
> - proposed.deps.json - here we specify list of PRs we want to commit
> atomically
> - override.deps.json - local file outside of version control which we
> consult in order to include development tools specific to contributor (code
> reloader, debugger, tracer, profiler, binpp, ...)
>
> Bellow is the example of a content of these files:
>
> ## deps.json
> {
> "src/b64url": [
> "https://github.com/apache/couchdb-b64url",
> "6895652d80f95cdf04efb14625abed868998f174"
> ],
> "src/cassim": [
> "https://github.com/apache/couchdb-cassim",
> "9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
> ],
> "src/chttpd": [
> "https://github.com/apache/couchdb-chttpd",
> "54e8f6147486d9afc5245e0143d15a4dd1185654"
> ],
> "src/meck": [
> "https://github.com/apache/couchdb-meck",
> "tree/0.8.2"
> ],
> ....
> }
>
> ## proposed.deps.json
> {
> "src/couch": "https://github.com/apache/couchdb-couch/pull/124",
> "src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"
> "src/couch_tests": [
> "https://github.com/apache/couchdb-erlang-tests",
> "tree/branch"
> ],
> }
>
> # Interface
>
> I propose to write a simple CLI tool to work with this structure. Bellow is
> a list of commands which we need to support (for minimal version)
>
> ## Adding new dependency
>
> git propose add https://github.com/apache/couchdb-foo
> a2d5ad2eedc960248b806f61df0a1009462bdb46
> git propose add https://github.com/apache/couchdb-bar tree/branch_name
>
> ## Adding new PR to the change set
>
> git propose add https://github.com/apache/couchdb-config/pull/4
>
> ## Checking out right dependencies
>
> git propose checkout
>
> ## Checking out release
>
> git propose checkout --release # this would ignore proposed.deps.json if it
> exists
>
> ## Merge the change
>
> This command would do the following:
> - Parse proposed.deps.json
> - Retrieve merge commit sha for every PR (exit if dependency is not merged
> yet)
> - Update dependencies in deps.json with correct merge commit sha
> - remove proposed.deps.json
>
> # Workflow
>
> export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv`
> git checkout -b feature-ZZZ
> cd src/X && hack dependency X
> cd ../..
> cd src/Y && hack dependency Y
> issue PRs for X and Y
> cd ../..
> git propose add https://github.com/apache/couchdb-X/pull/4
> git propose add https://github.com/apache/couchdb-Y/pull/49
> git add proposed.deps.json
> git commit -m "Commit feature {something} which does {a thing} and can be
> tested as {procedure}"
> git push origin feature-ZZZ
> ^ this would trigger our CI
> CI would do
> git propose checkout && ./configure && make check
>
> # Pros and Cons
>
> ## Pros
>
> - Changes are merged atomically
> - CI runs against expected versions of deps
> - Enables git bisect
> - Reduce tasks that needs to be done by ASF committer (no need to update
> rebar.config.script manually)
> - Simplifies testing of PRs by reviewers
> - Simplifies rebar.config since rebar is not used for managing deps
>
> ## Cons
>
> - some github.com specifics (concept of PRs and access to github API to get
> info about PR)
> - we need to have github.com as one of the remotes
> - we trigger CI only on push to main repository
>
> # Implementation
>
> We write a git-propose script in python and place it in ./bin. We add ./bin
> into either GIT_EXEC_PATH or PATH. You always can call the script directly
> (as ./bin/git-propose) if you don't like amending
> your environment.
>
> # Later improvements
>
> - We can issue PRs from the tool itself
> - We can merge from the tool itself
> - We can implement support for multiple remotes (asf, github, private)
> - We can implement support for multiple git transports (for the first
> version all repositories in *.deps.json files would use https://)
>
> Sincerely,
> ILYA KHLOPOTOV