You are viewing a plain text version of this content. The canonical link for it is here.
Posted to infrastructure-dev@apache.org by Paul Querna <pa...@querna.org> on 2010/11/03 15:08:03 UTC
Full support for Git at Apache, a dangerously indigo plan.
Please provide feedback and more tasks. This is a brainstorm. Please help out.
If you want to help with a task, go do it. The velocity of this
project is entirely dependent on volunteers.
Tomorrow I will take all the reasonable tasks we have, and put them
each into a INFRA jira ticket.
Stage -1: Please avoid bike sheds.
Goal: Indigo is my color today. <http://indigo.bikeshed.com/> We
will be providing full r/w git support for projects. It isn't a
trivial project, both for communities and for infrastructure.
Task: Recruit people to help with the following tasks.
Stage 0: Basic development and design
Goal: Resolve any remaining questions about design and
implementation of full Git support at Apache.
Task: Send this email. Status: Done!
Task: Evaluate Gitolite <https://github.com/sitaramc/gitolite> We
can definitely at least learn things from Gitolite even if we do not
end up using the code.
Task: Decide on push transport. Currently leaning towards offering
only HTTPS based pushing, as then it is very easy to re-use our
existing LDAP infrastructure. We can do SSH based with a single git
user + ssh public keys, but it is more work.
Task: Start putting tools, scripts and documentation in SVN.
<https://svn.apache.org/repos/infra/infrastructure/trunk/projects/git/>
Stage 1: Testing Repository
Goal: Build experience and document how the ASF is hosting
repositories, and ensure correct permissions, hooks, backups,
replication/mirror, etc are all working.
Task: Find server to do testing repository. My preference is to use
eris / harmonia as a pair in the long run, but for stage 1 we can just
use a pair of small VMs.
Task: HTTPS Server setup. Configure Apache + LDAP +
git-http-backend
<http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html>
Task: Figure out details of Git Slave. Just like SVN, we will have
git mirrored in both EU and US. A push to the EU server should
transparently push to the US server, and replicate back as needed.
Task: pre-receive hook: Authorization. Based on groups in LDAP,
restrict write access to members of the correct project. We will not
have path based authorization under, only whole project.
Task: pre-receive hook: Repository temporarily disabled. If
/etc/nocommit is present, do not allow any commits. This allows for
server and repository maintenance.
Task: pre-receive hook: Do not allow history rewriting / git push --force.
Task: post-update hook: Logging of commit. Log both the Apache
committer, and the 'git' committer. Figure out how to make this
information useful.
Task: post-receive hook: E-Mail notification. We have a traditional
dependence on 'good' commit emails. We need a very good mailer, as
SvnMailer handles many cases like large commits, weird charactersets,
etc, that most git based mailers don't even approach.
Task: post-receive hook: CIA Notification. <http://cia.vc/>
Task: post-receive hook: SvnPubSub Notification. I think we should
add support to SvnPubSub for git repositories. Maybe rename SvnPubSub
-> VcPubSub too.
Task: post-receive hook: Slave Mirroring. Every commit will be
mirrored to a second ASF machine, just like we do with Subversion.
This should just be a git push --mirror to a remote repository, but it
needs testing.
Task: post-receive hook: Github mirroring. After each push, every
repository will be mirrored to github.
Task: Find web based view of Git repositories. Gitweb or cgit or
viewvc. Evaluate and configure one of them.
Task: Determine policies for merge commits. See what Postgres has
done: <http://lwn.net/Articles/409635/>
Task: Document committer workflow. Needs a /dev/ page describing
all of the basic workflows that will be used by ASF committers. This
includes community expectations on how often to push, merge, and
moving patches around.
Task: Document infrastructure workflows. How to import a project
from SVN. How to import an external git repository. How to backup a
repository. How to verify a repository. How to setup a new TLP.
Task: Infrastructure Git run book. Add run book for all services
used in the git project.
<https://svn.apache.org/repos/infra/infrastructure/trunk/docs/services/>
Task: Consider SVN-Mirroring of projects. Once the primary VC is in
Git, it is possible to provide a replicated copy in Subversion, so
that existing users and build tools can continue using their checkouts
over SVN. We need to determine how feasible this is to actually
implement.
Stage 2: "beta" Testing by a project.
Goal: Test the new service on a volunteer project, ensuring both the
technical and community issues are successfully addressed.
Task: Testing full history conversion. We will be preserving 100%
of the history of a project.
Task: Evaluate after a few months. Whats working, whats not.
Iterate on problems and documentation.
Task: Ensure widespread Infrastructure Team knowledge of git. As
the goal is to have Git be a core service, equivalent to Subversion,
we must have multiple members of the team who are experienced with our
setup and git in general. "Team" includes adding new root@ members, so
step up :-)
Stage 3: Wide availability
Goal: Projects can choose Subversion or Git as their primary Version
Control, and conversion between each is done via standard INFRA
tickets.
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Tony Stevenson <to...@pc-tony.com>.
On Wed, Nov 03, 2010 at 10:08:03AM -0400, Paul Querna wrote:
> Please provide feedback and more tasks. This is a brainstorm. Please help out.
Added to SVN. infra/trunk/projects/git/
>
> If you want to help with a task, go do it. The velocity of this
> project is entirely dependent on volunteers.
>
> Tomorrow I will take all the reasonable tasks we have, and put them
> each into a INFRA jira ticket.
>
>
> Stage -1: Please avoid bike sheds.
>
> Goal: Indigo is my color today. <http://indigo.bikeshed.com/> We
> will be providing full r/w git support for projects. It isn't a
> trivial project, both for communities and for infrastructure.
>
> Task: Recruit people to help with the following tasks.
>
>
>
> Stage 0: Basic development and design
>
> Goal: Resolve any remaining questions about design and
> implementation of full Git support at Apache.
>
> Task: Send this email. Status: Done!
>
> Task: Evaluate Gitolite <https://github.com/sitaramc/gitolite> We
> can definitely at least learn things from Gitolite even if we do not
> end up using the code.
>
> Task: Decide on push transport. Currently leaning towards offering
> only HTTPS based pushing, as then it is very easy to re-use our
> existing LDAP infrastructure. We can do SSH based with a single git
> user + ssh public keys, but it is more work.
>
> Task: Start putting tools, scripts and documentation in SVN.
> <https://svn.apache.org/repos/infra/infrastructure/trunk/projects/git/>
>
>
>
> Stage 1: Testing Repository
>
> Goal: Build experience and document how the ASF is hosting
> repositories, and ensure correct permissions, hooks, backups,
> replication/mirror, etc are all working.
>
> Task: Find server to do testing repository. My preference is to use
> eris / harmonia as a pair in the long run, but for stage 1 we can just
> use a pair of small VMs.
>
> Task: HTTPS Server setup. Configure Apache + LDAP +
> git-http-backend
> <http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html>
>
> Task: Figure out details of Git Slave. Just like SVN, we will have
> git mirrored in both EU and US. A push to the EU server should
> transparently push to the US server, and replicate back as needed.
>
> Task: pre-receive hook: Authorization. Based on groups in LDAP,
> restrict write access to members of the correct project. We will not
> have path based authorization under, only whole project.
>
> Task: pre-receive hook: Repository temporarily disabled. If
> /etc/nocommit is present, do not allow any commits. This allows for
> server and repository maintenance.
>
> Task: pre-receive hook: Do not allow history rewriting / git push --force.
>
> Task: post-update hook: Logging of commit. Log both the Apache
> committer, and the 'git' committer. Figure out how to make this
> information useful.
>
> Task: post-receive hook: E-Mail notification. We have a traditional
> dependence on 'good' commit emails. We need a very good mailer, as
> SvnMailer handles many cases like large commits, weird charactersets,
> etc, that most git based mailers don't even approach.
>
> Task: post-receive hook: CIA Notification. <http://cia.vc/>
>
> Task: post-receive hook: SvnPubSub Notification. I think we should
> add support to SvnPubSub for git repositories. Maybe rename SvnPubSub
> -> VcPubSub too.
>
> Task: post-receive hook: Slave Mirroring. Every commit will be
> mirrored to a second ASF machine, just like we do with Subversion.
> This should just be a git push --mirror to a remote repository, but it
> needs testing.
>
> Task: post-receive hook: Github mirroring. After each push, every
> repository will be mirrored to github.
>
> Task: Find web based view of Git repositories. Gitweb or cgit or
> viewvc. Evaluate and configure one of them.
>
> Task: Determine policies for merge commits. See what Postgres has
> done: <http://lwn.net/Articles/409635/>
>
> Task: Document committer workflow. Needs a /dev/ page describing
> all of the basic workflows that will be used by ASF committers. This
> includes community expectations on how often to push, merge, and
> moving patches around.
>
> Task: Document infrastructure workflows. How to import a project
> from SVN. How to import an external git repository. How to backup a
> repository. How to verify a repository. How to setup a new TLP.
>
> Task: Infrastructure Git run book. Add run book for all services
> used in the git project.
> <https://svn.apache.org/repos/infra/infrastructure/trunk/docs/services/>
>
> Task: Consider SVN-Mirroring of projects. Once the primary VC is in
> Git, it is possible to provide a replicated copy in Subversion, so
> that existing users and build tools can continue using their checkouts
> over SVN. We need to determine how feasible this is to actually
> implement.
>
>
>
> Stage 2: "beta" Testing by a project.
>
> Goal: Test the new service on a volunteer project, ensuring both the
> technical and community issues are successfully addressed.
>
> Task: Testing full history conversion. We will be preserving 100%
> of the history of a project.
>
> Task: Evaluate after a few months. Whats working, whats not.
> Iterate on problems and documentation.
>
> Task: Ensure widespread Infrastructure Team knowledge of git. As
> the goal is to have Git be a core service, equivalent to Subversion,
> we must have multiple members of the team who are experienced with our
> setup and git in general. "Team" includes adding new root@ members, so
> step up :-)
>
>
>
> Stage 3: Wide availability
>
> Goal: Projects can choose Subversion or Git as their primary Version
> Control, and conversion between each is done via standard INFRA
> tickets.
--
Cheers,
Tony
Why, when no honest man will deny in private that every ultimate problem is
wrapped in the profoundest mystery, do honest men proclaim in pulpits that
unhesitating certainty is the duty of the most foolish and ignorant? Is it
not a spectacle to make the angels laugh? We are a company of ignorant
beings, feeling our way through mists and darkness, learning only be
incessantly repeated blunders, obtaining a glimmering of truth by falling
into every conceivable error, dimly discerning light enough for our daily
needs, but hopelessly differing whenever we attempt to describe the ultimate
origin or end of our paths; and yet, when one of us ventures to declare that
we don't know the map of the universe as well as the map of our infintesimal
parish, he is hooted, reviled, and perhaps told that he will be damned to all
eternity for his faithlessness.
-- Leslie Stephen, "An Agnostic's Apology",
Fortnightly Review, 1876
---------------------------------------
Tony Stevenson
tony@pc-tony.com // pctony@apache.org
tony@caret.cam.ac.uk
http://blog.pc-tony.com
GPG - 1024D/51047D66
--------------------------------------
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
On 11/3/2010 10:08 AM, Paul Querna wrote:
> <https://svn.apache.org/repos/infra/infrastructure/trunk/projects/git/>
I think we should commit this mail to the above and edit it live.
--
------------------------------------------------------------------------
1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C
Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354
VP Apache Infrastructure; Member, Apache Software Foundation
Committer, FreeBSD Foundation
Consultant, P6M7G8 Inc.
Sr. System Admin, Ridecharge Inc.
Work like you don't need the money,
love like you'll never get hurt,
and dance like nobody's watching.
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Thomas Koch <th...@koch.ro>.
Tony Stevenson:
> Theo Schnas-whatsa-nagle suggested we looked at mtrack [1] - He said it
> offerred all the features of github, and possibly some more, but the
> biggest benfit for us would be the ability to control the tooling.
>
> This is based on trac, but before you get all hot under th collar, take a
> look at the features.
>
> [1] -- http://groups.google.com/group/mtrack/
- mtrack is based on PHP so since we're all adults this should be reason
enough to avoid it.
- mtrack is mainly about issue tracking and project management, not a GIT repo
manager
Thomas Koch, http://www.koch.ro
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Tony Stevenson <to...@pc-tony.com>.
On Fri, Nov 05, 2010 at 10:00:11PM +0200, Jukka Zitting wrote:
> Hi,
>
> Finally got around to reacting to this. Great stuff, thanks for taking the lead!
>
> On Wed, Nov 3, 2010 at 4:08 PM, Paul Querna <pa...@querna.org> wrote:
> > Indigo is my color today. <http://indigo.bikeshed.com/>
>
> Did you consider pink, with yellow dots?
>
> > Task: Recruit people to help with the following tasks.
>
> Sign me up.
>
> > Task: Evaluate Gitolite <https://github.com/sitaramc/gitolite> We
> > can definitely at least learn things from Gitolite even if we do not
> > end up using the code.
>
> Gitolite looks interesting. I looked at Gitosis before, but dropped it
> since there was no obvious way to make it meet our legal/audit
> requirements. So far I've been looking at Gerrit, but perhaps Gitolite
> supports our needs too.
>
> The nice thing with Gerrit is that it's already used by Eclipse, who
> are quite similar to us, so we could just duplicate their setup. The
> bad thing with Gerrit is that it's Java software that Infra hasn't
> traditionally been too excited about.
Theo Schnas-whatsa-nagle suggested we looked at mtrack [1] - He said it offerred all the features of github, and possibly some more, but the biggest benfit for us would be the ability to control the tooling.
This is based on trac, but before you get all hot under th collar, take a look at the features.
[1] -- http://groups.google.com/group/mtrack/
>
> More comments in the INFRA issues you filed.
>
> BR,
>
> Jukka Zitting
--
Cheers,
Tony
---------------------------------------
Tony Stevenson
tony@pc-tony.com // pctony@apache.org
tony@caret.cam.ac.uk
http://blog.pc-tony.com
GPG - 1024D/51047D66
--------------------------------------
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
Finally got around to reacting to this. Great stuff, thanks for taking the lead!
On Wed, Nov 3, 2010 at 4:08 PM, Paul Querna <pa...@querna.org> wrote:
> Indigo is my color today. <http://indigo.bikeshed.com/>
Did you consider pink, with yellow dots?
> Task: Recruit people to help with the following tasks.
Sign me up.
> Task: Evaluate Gitolite <https://github.com/sitaramc/gitolite> We
> can definitely at least learn things from Gitolite even if we do not
> end up using the code.
Gitolite looks interesting. I looked at Gitosis before, but dropped it
since there was no obvious way to make it meet our legal/audit
requirements. So far I've been looking at Gerrit, but perhaps Gitolite
supports our needs too.
The nice thing with Gerrit is that it's already used by Eclipse, who
are quite similar to us, so we could just duplicate their setup. The
bad thing with Gerrit is that it's Java software that Infra hasn't
traditionally been too excited about.
More comments in the INFRA issues you filed.
BR,
Jukka Zitting
Use Gerrit? was: Full support for Git at Apache, a dangerously indigo plan.
Posted by Thomas Koch <th...@koch.ro>.
Hi,
please also consider gerrit[1]. Gerrit has recently been introduced at the
Eclipse Foundation[2]. It's a code review and IP tracking application
developed by google for the android development[3]. A gerrit test deployment
has already been done[4] by Jukka Zitting in February on git.a.o[5], but
doesn't seem to work right now.
Advantages of gerrit:
- enforcement and tracking of IP (both android and eclipse require this)
- enforcement of a work-flow (e.g. how many reviewer approvals, hudson
approval)
- already adopted at android, eclipse, ???
- LDAP integration is a standard feature
- AFAIR, there's replication build in?
There's a nice FLOSS weekly pod cast about gerrit[6].
I'll be full time student again from march '11 and could help out with gerrit
maintenance then.
[1] http://code.google.com/p/gerrit/
[2] https://bugs.eclipse.org/bugs/show_bug.cgi?id=283749
[3] http://source.android.com/source/life-of-a-patch.html
[4] https://issues.apache.org/jira/browse/INFRA-2205
[5] http://git.apache.org:8080/
[6] http://twit.tv/floss118
Thomas Koch, http://www.koch.ro
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Mark Thomas <ma...@apache.org>.
On 04/11/2010 00:13, Paul Querna wrote:
> On Thu, Nov 4, 2010 at 12:09 AM, Jeremy Thomerson
>> 1 - originally when I heard of full git read/write support, the requirement
>> was going to be that all git commits were still going to be pushed into SVN
>> as the "authoritative" backend. Is that the case, or will there be
>> "authoritative" git repos?
>
> Correct, this is authoritative repositories in Git.
A question for the Git gurus.
There are many projects at Apache that have dependencies on other ASF
projects. In most cases these dependencies are on the other project's
release artifacts so the choice of VCS has no impact. However, there are
some cases where a binary dependency is not the best solution and one
project uses a copy of the source from another project. In this case the
choice of VCS may have an impact.
One of the advantages of the current "everything in a single svn
repository" is that if one project takes a copy of some (or all) of the
source code from another project then a) the copy retains the history
and b) the copy can be updated with changes from the original with a
simple "svn merge ...". Both retaining the history and the ease of
keeping them in sync is important.
I appreciate that this might not happen often, but how would this work
in terms of retaining history and keeping the copy in sync) if:
- the source and destination projects are both in separate git repositories?
- the source project is in svn but the destination project is in git?
- the source project is in git but the destination project is in svn?
Cheers,
Mark
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Jeremy Thomerson <je...@thomersonfamily.com>.
On Wed, Nov 3, 2010 at 11:13 PM, Paul Querna <pa...@querna.org> wrote:
> On Thu, Nov 4, 2010 at 12:09 AM, Jeremy Thomerson
> <je...@thomersonfamily.com> wrote:
> > On Wed, Nov 3, 2010 at 9:08 AM, Paul Querna <pa...@querna.org> wrote:
> >
> >> Goal: Indigo is my color today. <http://indigo.bikeshed.com/> We
> >> will be providing full r/w git support for projects. It isn't a
> >> trivial project, both for communities and for infrastructure.
> >>
> >
> > This is great news! I unfortunately missed ApacheCon (and subsequently,
> > involvement in or listening in on these discussions) this year because of
> > client requirements. So, I have two follow-up questions:
> >
> > 1 - originally when I heard of full git read/write support, the
> requirement
> > was going to be that all git commits were still going to be pushed into
> SVN
> > as the "authoritative" backend. Is that the case, or will there be
> > "authoritative" git repos?
>
> Correct, this is authoritative repositories in Git.
>
Awesome. I'll help as much as time allows. Requesting git support is one
of the things that drove me to start lurking in infra stuff, because I knew
I couldn't request it and expect someone else to do all the work. I want to
contribute.
> > 2 - I had starting working on scripts to convert our read-only git access
> > into svnpubsub-based git syncing, and when I had that ready, I was going
> to
> > use it to try to get GitHub permission for us to push our git repos to
> them
> > rather than them pulling (since they sometimes fall days out of sync. Is
> > this still useful? I suppose it could be for making projects that will
> stay
> > svn-based available in read-only git format. But I want to be sure
> before I
> > continue.
>
> Yes, this is still useful. That is a very good improvement regardless.
>
> I suspect on the Github side, we should just request to take over the
> github.com/apache account, and turn it into a proper organization ran
> by the infrastructure team, so we can do the post-commit syncing, both
> for svn and git repositories.
>
Okay, I'll continue with that, although I've been swamped with
Wicket-related consulting, and not had much spare time lately. Maybe in
November.
Jeremy
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Paul Querna <pa...@querna.org>.
On Thu, Nov 4, 2010 at 12:09 AM, Jeremy Thomerson
<je...@thomersonfamily.com> wrote:
> On Wed, Nov 3, 2010 at 9:08 AM, Paul Querna <pa...@querna.org> wrote:
>
>> Goal: Indigo is my color today. <http://indigo.bikeshed.com/> We
>> will be providing full r/w git support for projects. It isn't a
>> trivial project, both for communities and for infrastructure.
>>
>
> This is great news! I unfortunately missed ApacheCon (and subsequently,
> involvement in or listening in on these discussions) this year because of
> client requirements. So, I have two follow-up questions:
>
> 1 - originally when I heard of full git read/write support, the requirement
> was going to be that all git commits were still going to be pushed into SVN
> as the "authoritative" backend. Is that the case, or will there be
> "authoritative" git repos?
Correct, this is authoritative repositories in Git.
> 2 - I had starting working on scripts to convert our read-only git access
> into svnpubsub-based git syncing, and when I had that ready, I was going to
> use it to try to get GitHub permission for us to push our git repos to them
> rather than them pulling (since they sometimes fall days out of sync. Is
> this still useful? I suppose it could be for making projects that will stay
> svn-based available in read-only git format. But I want to be sure before I
> continue.
Yes, this is still useful. That is a very good improvement regardless.
I suspect on the Github side, we should just request to take over the
github.com/apache account, and turn it into a proper organization ran
by the infrastructure team, so we can do the post-commit syncing, both
for svn and git repositories.
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Jeremy Thomerson <je...@thomersonfamily.com>.
On Wed, Nov 3, 2010 at 9:08 AM, Paul Querna <pa...@querna.org> wrote:
> Goal: Indigo is my color today. <http://indigo.bikeshed.com/> We
> will be providing full r/w git support for projects. It isn't a
> trivial project, both for communities and for infrastructure.
>
This is great news! I unfortunately missed ApacheCon (and subsequently,
involvement in or listening in on these discussions) this year because of
client requirements. So, I have two follow-up questions:
1 - originally when I heard of full git read/write support, the requirement
was going to be that all git commits were still going to be pushed into SVN
as the "authoritative" backend. Is that the case, or will there be
"authoritative" git repos?
2 - I had starting working on scripts to convert our read-only git access
into svnpubsub-based git syncing, and when I had that ready, I was going to
use it to try to get GitHub permission for us to push our git repos to them
rather than them pulling (since they sometimes fall days out of sync. Is
this still useful? I suppose it could be for making projects that will stay
svn-based available in read-only git format. But I want to be sure before I
continue.
Thanks!
Jeremy Thomerson
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Tony Stevenson <to...@pc-tony.com>.
This is a great start Paul. Thanks.
I'll re-read it later, when I dont have someone taling quite so loudly, and make more specific comments.
I'll create a category in INFRA for this, too
On Wed, Nov 03, 2010 at 10:08:03AM -0400, Paul Querna wrote:
> Please provide feedback and more tasks. This is a brainstorm. Please help out.
>
> If you want to help with a task, go do it. The velocity of this
> project is entirely dependent on volunteers.
>
> Tomorrow I will take all the reasonable tasks we have, and put them
> each into a INFRA jira ticket.
>
>
> Stage -1: Please avoid bike sheds.
>
> Goal: Indigo is my color today. <http://indigo.bikeshed.com/> We
> will be providing full r/w git support for projects. It isn't a
> trivial project, both for communities and for infrastructure.
>
> Task: Recruit people to help with the following tasks.
>
>
>
> Stage 0: Basic development and design
>
> Goal: Resolve any remaining questions about design and
> implementation of full Git support at Apache.
>
> Task: Send this email. Status: Done!
>
> Task: Evaluate Gitolite <https://github.com/sitaramc/gitolite> We
> can definitely at least learn things from Gitolite even if we do not
> end up using the code.
>
> Task: Decide on push transport. Currently leaning towards offering
> only HTTPS based pushing, as then it is very easy to re-use our
> existing LDAP infrastructure. We can do SSH based with a single git
> user + ssh public keys, but it is more work.
>
> Task: Start putting tools, scripts and documentation in SVN.
> <https://svn.apache.org/repos/infra/infrastructure/trunk/projects/git/>
>
>
>
> Stage 1: Testing Repository
>
> Goal: Build experience and document how the ASF is hosting
> repositories, and ensure correct permissions, hooks, backups,
> replication/mirror, etc are all working.
>
> Task: Find server to do testing repository. My preference is to use
> eris / harmonia as a pair in the long run, but for stage 1 we can just
> use a pair of small VMs.
>
> Task: HTTPS Server setup. Configure Apache + LDAP +
> git-http-backend
> <http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html>
>
> Task: Figure out details of Git Slave. Just like SVN, we will have
> git mirrored in both EU and US. A push to the EU server should
> transparently push to the US server, and replicate back as needed.
>
> Task: pre-receive hook: Authorization. Based on groups in LDAP,
> restrict write access to members of the correct project. We will not
> have path based authorization under, only whole project.
>
> Task: pre-receive hook: Repository temporarily disabled. If
> /etc/nocommit is present, do not allow any commits. This allows for
> server and repository maintenance.
>
> Task: pre-receive hook: Do not allow history rewriting / git push --force.
>
> Task: post-update hook: Logging of commit. Log both the Apache
> committer, and the 'git' committer. Figure out how to make this
> information useful.
>
> Task: post-receive hook: E-Mail notification. We have a traditional
> dependence on 'good' commit emails. We need a very good mailer, as
> SvnMailer handles many cases like large commits, weird charactersets,
> etc, that most git based mailers don't even approach.
>
> Task: post-receive hook: CIA Notification. <http://cia.vc/>
>
> Task: post-receive hook: SvnPubSub Notification. I think we should
> add support to SvnPubSub for git repositories. Maybe rename SvnPubSub
> -> VcPubSub too.
>
> Task: post-receive hook: Slave Mirroring. Every commit will be
> mirrored to a second ASF machine, just like we do with Subversion.
> This should just be a git push --mirror to a remote repository, but it
> needs testing.
>
> Task: post-receive hook: Github mirroring. After each push, every
> repository will be mirrored to github.
>
> Task: Find web based view of Git repositories. Gitweb or cgit or
> viewvc. Evaluate and configure one of them.
>
> Task: Determine policies for merge commits. See what Postgres has
> done: <http://lwn.net/Articles/409635/>
>
> Task: Document committer workflow. Needs a /dev/ page describing
> all of the basic workflows that will be used by ASF committers. This
> includes community expectations on how often to push, merge, and
> moving patches around.
>
> Task: Document infrastructure workflows. How to import a project
> from SVN. How to import an external git repository. How to backup a
> repository. How to verify a repository. How to setup a new TLP.
>
> Task: Infrastructure Git run book. Add run book for all services
> used in the git project.
> <https://svn.apache.org/repos/infra/infrastructure/trunk/docs/services/>
>
> Task: Consider SVN-Mirroring of projects. Once the primary VC is in
> Git, it is possible to provide a replicated copy in Subversion, so
> that existing users and build tools can continue using their checkouts
> over SVN. We need to determine how feasible this is to actually
> implement.
>
>
>
> Stage 2: "beta" Testing by a project.
>
> Goal: Test the new service on a volunteer project, ensuring both the
> technical and community issues are successfully addressed.
>
> Task: Testing full history conversion. We will be preserving 100%
> of the history of a project.
>
> Task: Evaluate after a few months. Whats working, whats not.
> Iterate on problems and documentation.
>
> Task: Ensure widespread Infrastructure Team knowledge of git. As
> the goal is to have Git be a core service, equivalent to Subversion,
> we must have multiple members of the team who are experienced with our
> setup and git in general. "Team" includes adding new root@ members, so
> step up :-)
>
>
>
> Stage 3: Wide availability
>
> Goal: Projects can choose Subversion or Git as their primary Version
> Control, and conversion between each is done via standard INFRA
> tickets.
--
Cheers,
Tony
If I have to lay an egg for my country, I'll do it.
-- Bob Hope
---------------------------------------
Tony Stevenson
tony@pc-tony.com // pctony@apache.org
tony@caret.cam.ac.uk
http://blog.pc-tony.com
GPG - 1024D/51047D66
--------------------------------------
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Emmanuel Lécharny <el...@apache.org>.
Holly cow !
On 11/3/10 3:08 PM, Paul Querna wrote:
> Task: pre-receive hook: Authorization. Based on groups in LDAP,
> restrict write access to members of the correct project. We will not
> have path based authorization under, only whole project.
We can (Directory) help when it comes to play with LDAP. Even if it's to
connect Git to OpenLDAP...
Great move !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Mark Struberg <st...@yahoo.de>.
A few such tools already exist, because that's what the Linux guys use also.
Another point: we have to be carefull with
git merge --squash and
git rebase -i
if we apply patches from external. Because this will cause loosing some IP information. Not sure if there is a way to omit this somehow, at least we need to make people sensitive for this area.
LieGrue,
strub
Not sure, but is there a way to
--- On Wed, 11/3/10, Brett Porter <br...@apache.org> wrote:
> From: Brett Porter <br...@apache.org>
> Subject: Re: Full support for Git at Apache, a dangerously indigo plan.
> To: infrastructure-dev@apache.org
> Date: Wednesday, November 3, 2010, 9:51 PM
> Probably a trivial thing, but how
> important are aggregated statistical info about where
> individual committers do their work, the "1,000,000 commits"
> type of tracking, svnsearch.org, etc. for us or PR? Do we
> need to do the svn commit mirroring to keep that, or perhaps
> have something else optionally available?
>
> On 03/11/2010, at 10:08 AM, Paul Querna wrote:
>
> > Please provide feedback and more tasks. This is
> a brainstorm. Please help out.
> >
> > If you want to help with a task, go do it. The
> velocity of this
> > project is entirely dependent on volunteers.
> >
> > Tomorrow I will take all the reasonable tasks we have,
> and put them
> > each into a INFRA jira ticket.
> >
> >
> > Stage -1: Please avoid bike sheds.
> >
> > Goal: Indigo is my color today. <http://indigo.bikeshed.com/> We
> > will be providing full r/w git support for
> projects. It isn't a
> > trivial project, both for communities and for
> infrastructure.
> >
> > Task: Recruit people to help with the following
> tasks.
> >
> >
> >
> > Stage 0: Basic development and design
> >
> > Goal: Resolve any remaining questions about
> design and
> > implementation of full Git support at Apache.
> >
> > Task: Send this email. Status: Done!
> >
> > Task: Evaluate Gitolite <https://github.com/sitaramc/gitolite> We
> > can definitely at least learn things from Gitolite
> even if we do not
> > end up using the code.
> >
> > Task: Decide on push transport. Currently
> leaning towards offering
> > only HTTPS based pushing, as then it is very easy to
> re-use our
> > existing LDAP infrastructure. We can do SSH
> based with a single git
> > user + ssh public keys, but it is more work.
> >
> > Task: Start putting tools, scripts and
> documentation in SVN.
> > <https://svn.apache.org/repos/infra/infrastructure/trunk/projects/git/>
> >
> >
> >
> > Stage 1: Testing Repository
> >
> > Goal: Build experience and document how the ASF
> is hosting
> > repositories, and ensure correct permissions, hooks,
> backups,
> > replication/mirror, etc are all working.
> >
> > Task: Find server to do testing
> repository. My preference is to use
> > eris / harmonia as a pair in the long run, but for
> stage 1 we can just
> > use a pair of small VMs.
> >
> > Task: HTTPS Server setup. Configure Apache
> + LDAP +
> > git-http-backend
> > <http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html>
> >
> > Task: Figure out details of Git Slave.
> Just like SVN, we will have
> > git mirrored in both EU and US. A push to the EU
> server should
> > transparently push to the US server, and replicate
> back as needed.
> >
> > Task: pre-receive hook: Authorization.
> Based on groups in LDAP,
> > restrict write access to members of the correct
> project. We will not
> > have path based authorization under, only whole
> project.
> >
> > Task: pre-receive hook: Repository temporarily
> disabled. If
> > /etc/nocommit is present, do not allow any
> commits. This allows for
> > server and repository maintenance.
> >
> > Task: pre-receive hook: Do not allow history
> rewriting / git push --force.
> >
> > Task: post-update hook: Logging of commit.
> Log both the Apache
> > committer, and the 'git' committer. Figure out how to
> make this
> > information useful.
> >
> > Task: post-receive hook: E-Mail
> notification. We have a traditional
> > dependence on 'good' commit emails. We need a
> very good mailer, as
> > SvnMailer handles many cases like large commits, weird
> charactersets,
> > etc, that most git based mailers don't even approach.
> >
> > Task: post-receive hook: CIA Notification.
> <http://cia.vc/>
> >
> > Task: post-receive hook: SvnPubSub
> Notification. I think we should
> > add support to SvnPubSub for git repositories. Maybe
> rename SvnPubSub
> > -> VcPubSub too.
> >
> > Task: post-receive hook: Slave Mirroring.
> Every commit will be
> > mirrored to a second ASF machine, just like we do with
> Subversion.
> > This should just be a git push --mirror to a remote
> repository, but it
> > needs testing.
> >
> > Task: post-receive hook: Github mirroring.
> After each push, every
> > repository will be mirrored to github.
> >
> > Task: Find web based view of Git
> repositories. Gitweb or cgit or
> > viewvc. Evaluate and configure one of them.
> >
> > Task: Determine policies for merge
> commits. See what Postgres has
> > done: <http://lwn.net/Articles/409635/>
> >
> > Task: Document committer workflow. Needs a
> /dev/ page describing
> > all of the basic workflows that will be used by ASF
> committers. This
> > includes community expectations on how often to push,
> merge, and
> > moving patches around.
> >
> > Task: Document infrastructure workflows.
> How to import a project
> > from SVN. How to import an external git
> repository. How to backup a
> > repository. How to verify a repository. How to
> setup a new TLP.
> >
> > Task: Infrastructure Git run book. Add run
> book for all services
> > used in the git project.
> > <https://svn.apache.org/repos/infra/infrastructure/trunk/docs/services/>
> >
> > Task: Consider SVN-Mirroring of projects.
> Once the primary VC is in
> > Git, it is possible to provide a replicated copy in
> Subversion, so
> > that existing users and build tools can continue using
> their checkouts
> > over SVN. We need to determine how feasible this
> is to actually
> > implement.
> >
> >
> >
> > Stage 2: "beta" Testing by a project.
> >
> > Goal: Test the new service on a volunteer
> project, ensuring both the
> > technical and community issues are successfully
> addressed.
> >
> > Task: Testing full history conversion. We
> will be preserving 100%
> > of the history of a project.
> >
> > Task: Evaluate after a few months. Whats
> working, whats not.
> > Iterate on problems and documentation.
> >
> > Task: Ensure widespread Infrastructure Team
> knowledge of git. As
> > the goal is to have Git be a core service, equivalent
> to Subversion,
> > we must have multiple members of the team who are
> experienced with our
> > setup and git in general. "Team" includes adding new
> root@ members, so
> > step up :-)
> >
> >
> >
> > Stage 3: Wide availability
> >
> > Goal: Projects can choose Subversion or Git as
> their primary Version
> > Control, and conversion between each is done via
> standard INFRA
> > tickets.
>
> --
> Brett Porter
> brett@apache.org
> http://brettporter.wordpress.com/
>
>
Re: Full support for Git at Apache, a dangerously indigo plan.
Posted by Brett Porter <br...@apache.org>.
Probably a trivial thing, but how important are aggregated statistical info about where individual committers do their work, the "1,000,000 commits" type of tracking, svnsearch.org, etc. for us or PR? Do we need to do the svn commit mirroring to keep that, or perhaps have something else optionally available?
On 03/11/2010, at 10:08 AM, Paul Querna wrote:
> Please provide feedback and more tasks. This is a brainstorm. Please help out.
>
> If you want to help with a task, go do it. The velocity of this
> project is entirely dependent on volunteers.
>
> Tomorrow I will take all the reasonable tasks we have, and put them
> each into a INFRA jira ticket.
>
>
> Stage -1: Please avoid bike sheds.
>
> Goal: Indigo is my color today. <http://indigo.bikeshed.com/> We
> will be providing full r/w git support for projects. It isn't a
> trivial project, both for communities and for infrastructure.
>
> Task: Recruit people to help with the following tasks.
>
>
>
> Stage 0: Basic development and design
>
> Goal: Resolve any remaining questions about design and
> implementation of full Git support at Apache.
>
> Task: Send this email. Status: Done!
>
> Task: Evaluate Gitolite <https://github.com/sitaramc/gitolite> We
> can definitely at least learn things from Gitolite even if we do not
> end up using the code.
>
> Task: Decide on push transport. Currently leaning towards offering
> only HTTPS based pushing, as then it is very easy to re-use our
> existing LDAP infrastructure. We can do SSH based with a single git
> user + ssh public keys, but it is more work.
>
> Task: Start putting tools, scripts and documentation in SVN.
> <https://svn.apache.org/repos/infra/infrastructure/trunk/projects/git/>
>
>
>
> Stage 1: Testing Repository
>
> Goal: Build experience and document how the ASF is hosting
> repositories, and ensure correct permissions, hooks, backups,
> replication/mirror, etc are all working.
>
> Task: Find server to do testing repository. My preference is to use
> eris / harmonia as a pair in the long run, but for stage 1 we can just
> use a pair of small VMs.
>
> Task: HTTPS Server setup. Configure Apache + LDAP +
> git-http-backend
> <http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html>
>
> Task: Figure out details of Git Slave. Just like SVN, we will have
> git mirrored in both EU and US. A push to the EU server should
> transparently push to the US server, and replicate back as needed.
>
> Task: pre-receive hook: Authorization. Based on groups in LDAP,
> restrict write access to members of the correct project. We will not
> have path based authorization under, only whole project.
>
> Task: pre-receive hook: Repository temporarily disabled. If
> /etc/nocommit is present, do not allow any commits. This allows for
> server and repository maintenance.
>
> Task: pre-receive hook: Do not allow history rewriting / git push --force.
>
> Task: post-update hook: Logging of commit. Log both the Apache
> committer, and the 'git' committer. Figure out how to make this
> information useful.
>
> Task: post-receive hook: E-Mail notification. We have a traditional
> dependence on 'good' commit emails. We need a very good mailer, as
> SvnMailer handles many cases like large commits, weird charactersets,
> etc, that most git based mailers don't even approach.
>
> Task: post-receive hook: CIA Notification. <http://cia.vc/>
>
> Task: post-receive hook: SvnPubSub Notification. I think we should
> add support to SvnPubSub for git repositories. Maybe rename SvnPubSub
> -> VcPubSub too.
>
> Task: post-receive hook: Slave Mirroring. Every commit will be
> mirrored to a second ASF machine, just like we do with Subversion.
> This should just be a git push --mirror to a remote repository, but it
> needs testing.
>
> Task: post-receive hook: Github mirroring. After each push, every
> repository will be mirrored to github.
>
> Task: Find web based view of Git repositories. Gitweb or cgit or
> viewvc. Evaluate and configure one of them.
>
> Task: Determine policies for merge commits. See what Postgres has
> done: <http://lwn.net/Articles/409635/>
>
> Task: Document committer workflow. Needs a /dev/ page describing
> all of the basic workflows that will be used by ASF committers. This
> includes community expectations on how often to push, merge, and
> moving patches around.
>
> Task: Document infrastructure workflows. How to import a project
> from SVN. How to import an external git repository. How to backup a
> repository. How to verify a repository. How to setup a new TLP.
>
> Task: Infrastructure Git run book. Add run book for all services
> used in the git project.
> <https://svn.apache.org/repos/infra/infrastructure/trunk/docs/services/>
>
> Task: Consider SVN-Mirroring of projects. Once the primary VC is in
> Git, it is possible to provide a replicated copy in Subversion, so
> that existing users and build tools can continue using their checkouts
> over SVN. We need to determine how feasible this is to actually
> implement.
>
>
>
> Stage 2: "beta" Testing by a project.
>
> Goal: Test the new service on a volunteer project, ensuring both the
> technical and community issues are successfully addressed.
>
> Task: Testing full history conversion. We will be preserving 100%
> of the history of a project.
>
> Task: Evaluate after a few months. Whats working, whats not.
> Iterate on problems and documentation.
>
> Task: Ensure widespread Infrastructure Team knowledge of git. As
> the goal is to have Git be a core service, equivalent to Subversion,
> we must have multiple members of the team who are experienced with our
> setup and git in general. "Team" includes adding new root@ members, so
> step up :-)
>
>
>
> Stage 3: Wide availability
>
> Goal: Projects can choose Subversion or Git as their primary Version
> Control, and conversion between each is done via standard INFRA
> tickets.
--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/