You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Dmitriy Lyubimov <dl...@gmail.com> on 2014/05/07 03:39:39 UTC
Exploring moving Mahout to git as main repo
Hi,
We are trying to explore an option of moving Mahout to GIT as its main
repository. What are the options to do this within current infrastucture?
How will we be able to do github pull requests similar to how Apache Spark
does it?
Thank you.
-Dmitriy
Re: Exploring moving Mahout to git as main repo
Posted by Ted Dunning <te...@gmail.com>.
Well, the system is becoming unmaintainable due to accumulated technical
debt. But infra has David Nalley in charge and he is leading a very
significant effort to pay down that debt and rebuild key services in a more
maintainable form. I have spoken at length with David and he really
impresses me as the right guy to make this all work better.
That said, he has a big task ahead and it won't happen in a moment. It
probably won't even be completely done in the next 24 months.
Regarding the suggestion to use out-sourced tools like Github, there is
established policy at Apache that critical information, notably including
the authoritative email record and source code have to be maintained on
Apache resources. Period. End of Discussion.
That doesn't mean that additional toolsets can be used to drive
productivity, but if it doesn't get decided on the mailing list, it didn't
get decided. This is critical to the community and longevity mission. The
recent problems are significant, but they will be resolved.
On Fri, May 16, 2014 at 4:35 PM, Pat Ferrel <pa...@gmail.com> wrote:
> Well since it’s come up (I guess I’m bringing it up)
>
> Maybe Apache is getting too big to manage all it’s own list servers (down
> for several groups last week), repos, bug trackers, wikis (cwiki looks like
> it’s down)
>
> Have we thought about Github (repo, bug tracking, Jenkins) + Google or
> Yahoo Groups + Stackoverflow?
> Replacing, er, most Apache infra? I’m super grateful for the guys who work
> on all this but is it becoming unmaintainable? With every new thing we ask
> for (git + svn integration? plus running git repos) they can probably do it
> but it adds more moving parts until some day...
>
>
> On May 16, 2014, at 3:37 PM, Ted Dunning <te...@gmail.com> wrote:
>
> On Fri, May 16, 2014 at 10:16 AM, Pat Ferrel <pa...@gmail.com> wrote:
>
> > Agree 100% but there is seems to be an extra level of indirection here.
> >
> > 1) do your work in your personal github repo, probably using a branch
> >
>
> Exactly.
>
>
> > 2) issue a pull request to ??? the github Mahout repo or Apache Mahout
> git
> > repo?
> >
>
> To the github Mahout mirror. Otherwise, yes.
>
>
> > 3) respond to the request somewhere to make sure the merge goes well.
> >
>
> exactly.
>
>
> > It would be nice if we only had to deal with github imo. They will
> trigger
> > a Jenkins build on repo changes.
> >
>
> Well... yes. It would be nice to use the pretty green button on the github
> site, but it isn't much more work to do the merges manual. For merges that
> are clean, there are 3 or 4 commands required to merge the PR and github
> provides them. Doing the JIRA is additional effort.
>
>
>
>
>
> >
> > On May 15, 2014, at 3:33 PM, Ted Dunning <te...@gmail.com> wrote:
> >
> > On Fri, May 9, 2014 at 4:17 PM, Pat Ferrel <pa...@occamsmachete.com>
> wrote:
> >
> >> But pull requests would just be for contributors, right? Committers
> > should
> >> be able to push to the master directly, right?
> >>
> >
> > Not necessarily. Pull requests are so lovely nice to work with no matter
> > who you are.
> >
> > I can easily imagine using them myself for a long-lived change.
> >
> >
>
>
Re: Exploring moving Mahout to git as main repo
Posted by Pat Ferrel <pa...@gmail.com>.
Well since it’s come up (I guess I’m bringing it up)
Maybe Apache is getting too big to manage all it’s own list servers (down for several groups last week), repos, bug trackers, wikis (cwiki looks like it’s down)
Have we thought about Github (repo, bug tracking, Jenkins) + Google or Yahoo Groups + Stackoverflow?
Replacing, er, most Apache infra? I’m super grateful for the guys who work on all this but is it becoming unmaintainable? With every new thing we ask for (git + svn integration? plus running git repos) they can probably do it but it adds more moving parts until some day...
On May 16, 2014, at 3:37 PM, Ted Dunning <te...@gmail.com> wrote:
On Fri, May 16, 2014 at 10:16 AM, Pat Ferrel <pa...@gmail.com> wrote:
> Agree 100% but there is seems to be an extra level of indirection here.
>
> 1) do your work in your personal github repo, probably using a branch
>
Exactly.
> 2) issue a pull request to ??? the github Mahout repo or Apache Mahout git
> repo?
>
To the github Mahout mirror. Otherwise, yes.
> 3) respond to the request somewhere to make sure the merge goes well.
>
exactly.
> It would be nice if we only had to deal with github imo. They will trigger
> a Jenkins build on repo changes.
>
Well... yes. It would be nice to use the pretty green button on the github
site, but it isn't much more work to do the merges manual. For merges that
are clean, there are 3 or 4 commands required to merge the PR and github
provides them. Doing the JIRA is additional effort.
>
> On May 15, 2014, at 3:33 PM, Ted Dunning <te...@gmail.com> wrote:
>
> On Fri, May 9, 2014 at 4:17 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
>> But pull requests would just be for contributors, right? Committers
> should
>> be able to push to the master directly, right?
>>
>
> Not necessarily. Pull requests are so lovely nice to work with no matter
> who you are.
>
> I can easily imagine using them myself for a long-lived change.
>
>
Re: Exploring moving Mahout to git as main repo
Posted by Ted Dunning <te...@gmail.com>.
On Fri, May 16, 2014 at 10:16 AM, Pat Ferrel <pa...@gmail.com> wrote:
> Agree 100% but there is seems to be an extra level of indirection here.
>
> 1) do your work in your personal github repo, probably using a branch
>
Exactly.
> 2) issue a pull request to ??? the github Mahout repo or Apache Mahout git
> repo?
>
To the github Mahout mirror. Otherwise, yes.
> 3) respond to the request somewhere to make sure the merge goes well.
>
exactly.
> It would be nice if we only had to deal with github imo. They will trigger
> a Jenkins build on repo changes.
>
Well... yes. It would be nice to use the pretty green button on the github
site, but it isn't much more work to do the merges manual. For merges that
are clean, there are 3 or 4 commands required to merge the PR and github
provides them. Doing the JIRA is additional effort.
>
> On May 15, 2014, at 3:33 PM, Ted Dunning <te...@gmail.com> wrote:
>
> On Fri, May 9, 2014 at 4:17 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> > But pull requests would just be for contributors, right? Committers
> should
> > be able to push to the master directly, right?
> >
>
> Not necessarily. Pull requests are so lovely nice to work with no matter
> who you are.
>
> I can easily imagine using them myself for a long-lived change.
>
>
Re: Exploring moving Mahout to git as main repo
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
On Fri, May 16, 2014 at 10:16 AM, Pat Ferrel <pa...@gmail.com> wrote:
> Agree 100% but there is seems to be an extra level of indirection here.
>
> 1) do your work in your personal github repo, probably using a branch
>
personal github branch, yes.
> 2) issue a pull request to ??? the github Mahout repo or Apache Mahout git
> repo?
>
I assume it will be apache/mahout on github. At least it is for Spark.
> 3) respond to the request somewhere to make sure the merge goes well.
>
a merge of PR is handled by github itself. At least if merge is possible by
recursive strategy. Actually it is an interesting detail here. I am not
sure I understand 100% how it works exactly, here, but i am sure we will
figure it out. most likely, like Jake said, we will have to merge it
locally and push it to git-wp.a.o with a message that would close the PR in
github. If I understand it correctly, that's one of the things they've
implemented in github integration
>
> It would be nice if we only had to deal with github imo. They will trigger
> a Jenkins build on repo changes.
>
> On May 15, 2014, at 3:33 PM, Ted Dunning <te...@gmail.com> wrote:
>
> On Fri, May 9, 2014 at 4:17 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> > But pull requests would just be for contributors, right? Committers
> should
> > be able to push to the master directly, right?
> >
>
> Not necessarily. Pull requests are so lovely nice to work with no matter
> who you are.
>
> I can easily imagine using them myself for a long-lived change.
>
>
Re: Exploring moving Mahout to git as main repo
Posted by Pat Ferrel <pa...@gmail.com>.
Agree 100% but there is seems to be an extra level of indirection here.
1) do your work in your personal github repo, probably using a branch
2) issue a pull request to ??? the github Mahout repo or Apache Mahout git repo?
3) respond to the request somewhere to make sure the merge goes well.
It would be nice if we only had to deal with github imo. They will trigger a Jenkins build on repo changes.
On May 15, 2014, at 3:33 PM, Ted Dunning <te...@gmail.com> wrote:
On Fri, May 9, 2014 at 4:17 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> But pull requests would just be for contributors, right? Committers should
> be able to push to the master directly, right?
>
Not necessarily. Pull requests are so lovely nice to work with no matter
who you are.
I can easily imagine using them myself for a long-lived change.
Re: Exploring moving Mahout to git as main repo
Posted by Ted Dunning <te...@gmail.com>.
On Fri, May 9, 2014 at 4:17 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> But pull requests would just be for contributors, right? Committers should
> be able to push to the master directly, right?
>
Not necessarily. Pull requests are so lovely nice to work with no matter
who you are.
I can easily imagine using them myself for a long-lived change.
Re: Exploring moving Mahout to git as main repo
Posted by tuxdna <tu...@gmail.com>.
+1 for Git repositories
+1 for GitHub pull requests
Regarding "github pull", on reading the the wiki [1], below is what I
would do, unless there is some automation involved.
As acommitter, I would first merge pull request on GitHub repo and
then push it onto Apache's Git repository.
Infrastructure team can provide better ways :-)
Regards,
Saleem
[1] https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
Re: Exploring moving Mahout to git as main repo
Posted by Andrew Musselman <an...@gmail.com>.
+1
On Fri, May 9, 2014 at 4:17 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> Yes! I mentioned this awhile back. Pull requests are just patches under
> the covers, just soo easy to create.
>
> But pull requests would just be for contributors, right? Committers should
> be able to push to the master directly, right?
>
> On May 6, 2014, at 6:39 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
> Hi,
>
> We are trying to explore an option of moving Mahout to GIT as its main
> repository. What are the options to do this within current infrastucture?
>
> How will we be able to do github pull requests similar to how Apache Spark
> does it?
>
> Thank you.
> -Dmitriy
>
>
Re: Exploring moving Mahout to git as main repo
Posted by Andrew Musselman <an...@gmail.com>.
Yes, this is a very important part of the workflow that we're having
trouble with/ignoring via reviewboard and svn.
On Thu, May 15, 2014 at 12:40 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
> Yes, and no.
>
> Yes, committers should be able to push, or, rather, merge pull requests.
> But in case committers are also acting as contributors, i'd expect a pull
> request to be filed, since it is more than just this. It is also a review
> board. I've been struggling with integration issues in apache review board,
> and it seems, not just me alone. Github reviews are nicely integrated, it
> is a nice way to collaborate. Instead of cluttering jira, we could just
> discuss details right on the PR and be very specific about points of code
> we are talking to.
>
>
> On Fri, May 9, 2014 at 4:17 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> > Yes! I mentioned this awhile back. Pull requests are just patches under
> > the covers, just soo easy to create.
> >
> > But pull requests would just be for contributors, right? Committers
> should
> > be able to push to the master directly, right?
> >
> > On May 6, 2014, at 6:39 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> >
> > Hi,
> >
> > We are trying to explore an option of moving Mahout to GIT as its main
> > repository. What are the options to do this within current infrastucture?
> >
> > How will we be able to do github pull requests similar to how Apache
> Spark
> > does it?
> >
> > Thank you.
> > -Dmitriy
> >
> >
>
Re: Exploring moving Mahout to git as main repo
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Yes, and no.
Yes, committers should be able to push, or, rather, merge pull requests.
But in case committers are also acting as contributors, i'd expect a pull
request to be filed, since it is more than just this. It is also a review
board. I've been struggling with integration issues in apache review board,
and it seems, not just me alone. Github reviews are nicely integrated, it
is a nice way to collaborate. Instead of cluttering jira, we could just
discuss details right on the PR and be very specific about points of code
we are talking to.
On Fri, May 9, 2014 at 4:17 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> Yes! I mentioned this awhile back. Pull requests are just patches under
> the covers, just soo easy to create.
>
> But pull requests would just be for contributors, right? Committers should
> be able to push to the master directly, right?
>
> On May 6, 2014, at 6:39 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
> Hi,
>
> We are trying to explore an option of moving Mahout to GIT as its main
> repository. What are the options to do this within current infrastucture?
>
> How will we be able to do github pull requests similar to how Apache Spark
> does it?
>
> Thank you.
> -Dmitriy
>
>
Re: Exploring moving Mahout to git as main repo
Posted by Pat Ferrel <pa...@occamsmachete.com>.
Yes! I mentioned this awhile back. Pull requests are just patches under the covers, just soo easy to create.
But pull requests would just be for contributors, right? Committers should be able to push to the master directly, right?
On May 6, 2014, at 6:39 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
Hi,
We are trying to explore an option of moving Mahout to GIT as its main
repository. What are the options to do this within current infrastucture?
How will we be able to do github pull requests similar to how Apache Spark
does it?
Thank you.
-Dmitriy