[PROPOSAL] Source code migration and process for committers and contributors

[Richard Downer <ri...@apache.org>]

Taking into account the various opinions expressed on this list and
other venues, I present a proposal for migrating the Brooklyn source
code onto Apache infrastructure, and the processes for committers and
contributors going forward.

THE PROPOSAL

The IPMC proposes that Apache Brooklyn will adopt a model for storing
source code, processes for committing change to the source code, and
processes for external contributors to contribute code, that is
substantially the same as those used by the Apache jclouds project
(referred to as "the jclouds model" henceforth).

The jclouds model is described in these links, for contributors and
committers respectively:

https://wiki.apache.org/jclouds/How%20to%20Contribute
https://wiki.apache.org/jclouds/Committers%20Guide

The key points of the jclouds model are:

- The repository of record, against which all commits are made, is the
one hosted at git-wip-us.apache.org
- The repository is mirrored by automated process to a repository at
GitHub, in an organisation exclusively reserved for its project - we
term this repository the "working mirror"
- Contributors are invited to submit pull requests on the working
mirror located at GitHub
- Committers will, once satisfied that the patch is of sufficient
quality and that the contributor has an ICLA on file, merge the patch
to the git-wip-us.apache.org repository

SPECIFICS FOR BROOKLYN

Prior to its move adoption by the Apache Incubator, the Brooklyn
source code was in a GitHub organisation named "brooklyncentral",
which is owned by Cloudsoft, in a repository named "brooklyn". This
repository will be used as a seed for the repository at
git-wip-us.apache.org.

The working mirror at GitHub will be in a new organisation named
"brooklyn", which will be owned by The Apache Software Foundation.
Therefore, the working mirror against which contributors can submit
pull requests will be located at https://github.com/brooklyn/brooklyn.

At the point of switchover, Cloudsoft (the owners of the
"brooklyncentral" organisation) will initiate a transfer of the
"brooklyn" repository to the "brooklyn" organisation (owned by
Apache). This transfer will preserve contributor's forks, pull
requests and other metadata.

Finally, the process for contributors and committers will be
documented in a suitable location (most likely on the
http://brooklyn.incubator.apache.org website)

SUPPLEMENTARY NOTES

There is considerable interest in following the model used by the
Usergrid project, which differs in that commits are made against a
GitHub repository which is mirrored by an automated process to
git-wip-us.apache.org. However we note that the Foundation's board and
our mentors have expressed doubt over that model. Should the Usergrid
model gain acceptance with board, we may make a proposal to replace
our policy with the Usergrid model.

Re: [PROPOSAL] Source code migration and process for committers and contributors

[Alex Heneveld <al...@cloudsoftcorp.com>]

Thanks for the guidance, David, and for the link to:

https://blogs.apache.org/infra/entry/improved_integration_between_apache_and

There are some nice features described there, including PR's going to 
mailing list and IRC.  +1 to using it.

Text could be along the lines of:

* Contributors should fork and make pull requests to github 
apache/incubator-brooklyn

* The canonical repository is at git-wip-us.apache.org .  This is 
mirrored to the above github.

* The github accounts "brooklyn" and "brooklyncentral" are maintained 
externally.
    They host ancillary blueprints and projects (non-ASF but Apache 
licensed)
    and a mirror of the main project for convenience and backwards 
compatibility.
    PR's for Apache Brooklyn should *not* be made against these accounts.

WDYT?

Best
Alex


On 01/06/2014 22:32, David Nalley wrote:
> On 06/01, Andrew Kennedy wrote:
>>> There are a lot of challenges in incubation; dealing with
>>> IP clearance, getting a first release out, and acclimating to the ASF
>>> itself. Trying to pioneer new workflows and navigate the politics is a
>>> distraction to the effort to become a TLP and to ship software
>> +1
>>
>> I think this is a very good point. I'd be happy to go along with whatever
>> Git workflow is preferred by the ASF, although in that case I think rather
>> than us trying to work out an acceptable process, the responsibility is on
>> Apache to define the process we should follow.
>>
>> So, Dave, Chip et al, is there a definitive statement anywhere we can refer
>> to that describes the acceptable best practice Git workflow, and we will
>> just follow that?
>>
> So http://git-wip-us.apache.org has some documentation, but it's almost certainly not what you are looking for.
> This is essentially one reason the problem has occurred. In the 'old days' when there was only CVS or SVN it wasn't a big deal, there was only one true repository; so as a result no documentation exists.
> In terms of 'acceptable' the key tenants for now are:
> 1. Treat the ASF repo as canonical. Committers should be doing their work there. It's easy to tell when this isn't the case; or when folks are slurping data back and forth between external repos.
> 2. If I were prescribing what you should do, I'd say if you want some form of github interaction - you can use the ASF-provided stuff detailed at:
> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
>
> This doesn't mean you can't do other things, but the above is the path of least resistance IMO.
>
> --David

Re: [PROPOSAL] Source code migration and process for committers and contributors

[David Nalley <da...@gnsa.us>]

On 06/01, Andrew Kennedy wrote:
> > There are a lot of challenges in incubation; dealing with
> > IP clearance, getting a first release out, and acclimating to the ASF
> > itself. Trying to pioneer new workflows and navigate the politics is a
> > distraction to the effort to become a TLP and to ship software
> 
> +1
> 
> I think this is a very good point. I'd be happy to go along with whatever
> Git workflow is preferred by the ASF, although in that case I think rather
> than us trying to work out an acceptable process, the responsibility is on
> Apache to define the process we should follow.
> 
> So, Dave, Chip et al, is there a definitive statement anywhere we can refer
> to that describes the acceptable best practice Git workflow, and we will
> just follow that?
> 

So http://git-wip-us.apache.org has some documentation, but it's almost certainly not what you are looking for. 
This is essentially one reason the problem has occurred. In the 'old days' when there was only CVS or SVN it wasn't a big deal, there was only one true repository; so as a result no documentation exists. 
In terms of 'acceptable' the key tenants for now are: 
1. Treat the ASF repo as canonical. Committers should be doing their work there. It's easy to tell when this isn't the case; or when folks are slurping data back and forth between external repos. 
2. If I were prescribing what you should do, I'd say if you want some form of github interaction - you can use the ASF-provided stuff detailed at: 
https://blogs.apache.org/infra/entry/improved_integration_between_apache_and

This doesn't mean you can't do other things, but the above is the path of least resistance IMO. 

--David

Re: [PROPOSAL] Source code migration and process for committers and contributors

[Andrew Kennedy <an...@cloudsoftcorp.com>]

> There are a lot of challenges in incubation; dealing with
> IP clearance, getting a first release out, and acclimating to the ASF
> itself. Trying to pioneer new workflows and navigate the politics is a
> distraction to the effort to become a TLP and to ship software

+1

I think this is a very good point. I'd be happy to go along with whatever
Git workflow is preferred by the ASF, although in that case I think rather
than us trying to work out an acceptable process, the responsibility is on
Apache to define the process we should follow.

So, Dave, Chip et al, is there a definitive statement anywhere we can refer
to that describes the acceptable best practice Git workflow, and we will
just follow that?

Thanks,
Andrew.
-- 
-- andrew kennedy ? cloudsoft & software engineer : @grkvlt ;

Re: [PROPOSAL] Source code migration and process for committers and contributors

[David Nalley <da...@gnsa.us>]

On Sat, May 31, 2014 at 8:01 AM, Richard Downer <ri...@apache.org> wrote:
> Taking into account the various opinions expressed on this list and
> other venues, I present a proposal for migrating the Brooklyn source
> code onto Apache infrastructure, and the processes for committers and
> contributors going forward.
>
> THE PROPOSAL
>
> The IPMC proposes that Apache Brooklyn will adopt a model for storing

s/IPMC/PPMC/

> source code, processes for committing change to the source code, and
> processes for external contributors to contribute code, that is
> substantially the same as those used by the Apache jclouds project
> (referred to as "the jclouds model" henceforth).

It's important to realize that you'll see many things in the ASF that
have been instructive, and remain in place, but aren't good precedent
for other projects. jclouds (and Deltacloud before them) evolved in
its current state because no internal infrastructure existed to
satisfy the Foundation's requirements of the project and the project's
desire for a github workflow. The existing github workflow that exists
at the ASF was born out of that. I suspect, at some point in the
future, that jclouds will end up abandoning their workflow since the
github workflow at the ASF is a bit more polished and will require
less maintenance over time.

>
> The jclouds model is described in these links, for contributors and
> committers respectively:
>
> https://wiki.apache.org/jclouds/How%20to%20Contribute
> https://wiki.apache.org/jclouds/Committers%20Guide
>
> The key points of the jclouds model are:
>
> - The repository of record, against which all commits are made, is the
> one hosted at git-wip-us.apache.org
> - The repository is mirrored by automated process to a repository at
> GitHub, in an organisation exclusively reserved for its project - we
> term this repository the "working mirror"
> - Contributors are invited to submit pull requests on the working
> mirror located at GitHub
> - Committers will, once satisfied that the patch is of sufficient
> quality and that the contributor has an ICLA on file, merge the patch
> to the git-wip-us.apache.org repository
>
> SPECIFICS FOR BROOKLYN
>
> Prior to its move adoption by the Apache Incubator, the Brooklyn
> source code was in a GitHub organisation named "brooklyncentral",
> which is owned by Cloudsoft, in a repository named "brooklyn". This
> repository will be used as a seed for the repository at
> git-wip-us.apache.org.
>
> The working mirror at GitHub will be in a new organisation named
> "brooklyn", which will be owned by The Apache Software Foundation.
> Therefore, the working mirror against which contributors can submit
> pull requests will be located at https://github.com/brooklyn/brooklyn.
>

The ASF respectfully declines the offer for this GH organization.
The ASF maintains the github.com/apache organization; and with my
Infra hat on, if the ASF were to assume ownership, I don't think you'd
see any benefit over having a mirror in the Apache organization.

> At the point of switchover, Cloudsoft (the owners of the
> "brooklyncentral" organisation) will initiate a transfer of the
> "brooklyn" repository to the "brooklyn" organisation (owned by
> Apache). This transfer will preserve contributor's forks, pull
> requests and other metadata.
>

There are IP issues with preserving the PR.
The pull requests will need to be reinitiated to ensure that they are
considered contributions to the project.
I actually have some concerns about the mirroring that's already been
taking place. There may be licensing agreements in place for
CloudSoft, but the blanket mirroring of any commit occurring in
CloudSoft's git repo being copied over to the ASF repo is problematic
and we need to decide how we expect to account for all of those
commits.



> Finally, the process for contributors and committers will be
> documented in a suitable location (most likely on the
> http://brooklyn.incubator.apache.org website)
>
> SUPPLEMENTARY NOTES
>
> There is considerable interest in following the model used by the
> Usergrid project, which differs in that commits are made against a
> GitHub repository which is mirrored by an automated process to
> git-wip-us.apache.org. However we note that the Foundation's board and
> our mentors have expressed doubt over that model. Should the Usergrid
> model gain acceptance with board, we may make a proposal to replace
> our policy with the Usergrid model.

The Usergrid model as it stands today has been found to be
unacceptable to the Board. The mechanics and record keeping of the
current model is unacceptable to Infrastructure. The IPMC may or may
not have a problem with it. That is a long term effort that (in my
opinion) is unlikely to be acceptable in anything close to its current
incarnation. There are a lot of challenges in incubation; dealing with
IP clearance, getting a first release out, and acclimating to the ASF
itself. Trying to pioneer new workflows and navigate the politics is a
distraction to the effort to become a TLP and to ship software.

The ASF doesn't have the guidelines it does for projects because we
believe that there is only one true way to get things done. (We'd
still be on SVN if that was the case) or because we are all
curmudgeonly and set in our ways (though a few of us are). The issue
is that the Foundation is expected to serve as a steward for billions
of dollars worth of IP, in some of the most established and up and
coming sectors in software. Fulfilling that role at the scale of
almost 200 active projects as well as scores of no-longer active
projects means that we aren't as flexible as a small project living on
github or code.google.com, and it's one of the tradeoffs.

--David