You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "MrAsanjar ." <af...@gmail.com> on 2016/05/12 05:45:42 UTC
Hadoop CI with alternate architectures.
I am writing this email to reduce mishaps similar to the issue reported by
JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell, an
x86 specific
performance enhancement broke Hadoop build on Power and SPARC architecture.
To avoid similar issues in future, could I offer my help here, as a
OpenPOWER foundation member.
For example, we could contribute a Power based Jenkins slave(s) to Apache
Hadoop CI. As we have successfully done similar contribution to Apache
Bigtop CI in past https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/.
Thus, we could catch any regressions earlier in Hadoop development cycle.
I'd appreciate community's guidance on this.
Re: Hadoop CI with alternate architectures.
Posted by Steve Loughran <st...@hortonworks.com>.
> On 18 May 2016, at 16:29, Allen Wittenauer <aw...@apache.org> wrote:
>
>
>
> That’s really a question for infrastructure-dev@apache.org . They manage the ASF build infrastructure which Apache Hadoop and lots of other projects utilize. (Bigtop uses something custom, which I think is funded by Cloudera.)
There's no dedicated Hadoop boxes, just a pool of machines for all projects. I don't know about bigtop.
There's a mailing list, builds@apache.org, for people who want to get involved
>
> Once it is registered with builds.apache.org, it’s just a matter of us enabling a Jenkins job for it. I suspect we’ll likely do it like we have done other architectures in the past: set up a nightly job that people will need to cognizant of. Adding it to precommit would likely be too much and/or require some significant Yetus work to support reports from multiple hosts.
>
Nightly would be a good initial step; the hard part is getting it stable enough for its build-failed messages to not get ignored (it should be on send-no-emails mode until stable). Then comes the cultural shift: getting people to care that the build is failing. That'll be from native, OS and JVM issues.
Maybe Yetus could be set to warn that on changes which go near the .C or makefile code, that someone should explicitly kick off a precommit run on the PPC. That could be enough to remind those people playing with native code that that's something that really needs a check
> FWIW:
>
> * We have a Mac mini which, ironically, I thought I’d see if I could get a nightly job running on while I type this.
> * We have a dedicated Windows box which appears to be completely screwed up.
the default state of all windows boxes, sadly. The VMs are odd too; the FS permissions are enough to stop MiniYarnClusters coming up.
> * In the past, I think it was IBM that hinted that they would be willing to give ASF a PowerPC box but that offer seems to have disappeared.
>
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
Not sure what else needs to be done to get any comments from apache build
team.
This is getting frustrating.
In a nutshell, we (OpenPOWER foundation) willing to contribute both
hardware and resources to enable Apache Hadoop CI on Power architecture as
well. Is there anyone there that could help?
On Wed, Jun 8, 2016 at 7:13 AM, MrAsanjar . <af...@gmail.com> wrote:
> Apache Build team, any comments in this regard would be greatly
> appreciated.
> https://issues.apache.org/jira/browse/INFRA-12014
>
> On Tue, May 31, 2016 at 7:28 PM, MrAsanjar . <af...@gmail.com> wrote:
>
>> Allen, I couldn't agree more with the opinions stated in your reply.
>> However it has to be done soon or later. We (OpenPower community) are
>> willing to contribute in anyway possible.
>>
>> On Tue, May 31, 2016 at 10:02 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>
>>>
>>> > On May 31, 2016, at 9:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>>> >
>>> > Thanks Allen for your advise,
>>> > We would have hoped that this problem could have been resolved by
>>> simply getting in touch with the community, the same way as we had done
>>> with Apache Bigtop. Their support has been superb.
>>>
>>> Apache Bigtop is but one project. Adding PPC gear to the ASF
>>> build pool has a significantly larger impact since a lot more of the Hadoop
>>> ecosystem and ASF project in general are sharing that infrastructure.
>>
>>
>>
>
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
Not sure what else needs to be done to get any comments from apache build
team.
This is getting frustrating.
In a nutshell, we (OpenPOWER foundation) willing to contribute both
hardware and resources to enable Apache Hadoop CI on Power architecture as
well. Is there anyone there that could help?
On Wed, Jun 8, 2016 at 7:13 AM, MrAsanjar . <af...@gmail.com> wrote:
> Apache Build team, any comments in this regard would be greatly
> appreciated.
> https://issues.apache.org/jira/browse/INFRA-12014
>
> On Tue, May 31, 2016 at 7:28 PM, MrAsanjar . <af...@gmail.com> wrote:
>
>> Allen, I couldn't agree more with the opinions stated in your reply.
>> However it has to be done soon or later. We (OpenPower community) are
>> willing to contribute in anyway possible.
>>
>> On Tue, May 31, 2016 at 10:02 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>
>>>
>>> > On May 31, 2016, at 9:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>>> >
>>> > Thanks Allen for your advise,
>>> > We would have hoped that this problem could have been resolved by
>>> simply getting in touch with the community, the same way as we had done
>>> with Apache Bigtop. Their support has been superb.
>>>
>>> Apache Bigtop is but one project. Adding PPC gear to the ASF
>>> build pool has a significantly larger impact since a lot more of the Hadoop
>>> ecosystem and ASF project in general are sharing that infrastructure.
>>
>>
>>
>
Re: Hadoop CI with alternate architectures.
Posted by Geoffrey Corey <co...@apache.org>.
He did create an INFRA ticket.
@MrAsanjar it has been noted and added to the queue. However, do note that
there are currently a small number of people to work on INFRA tasks, so
getting the time to work on your power added to builds cluster tasks might
be delayed.
On Jun 8, 2016 6:38 AM, "Upayavira" <uv...@odoko.co.uk> wrote:
I would suggest you either mail infrastructure@, or better, create an
INFRA ticket in JIRA. This list isn't used much anymore, and I suspect
isn't where the infrastructure team hang out.
Upayavira
On Wed, 8 Jun 2016, at 01:13 PM, MrAsanjar . wrote:
> Apache Build team, any comments in this regard would be greatly
> appreciated.
> https://issues.apache.org/jira/browse/INFRA-12014
>
> On Tue, May 31, 2016 at 7:28 PM, MrAsanjar . <af...@gmail.com> wrote:
>
> > Allen, I couldn't agree more with the opinions stated in your reply.
> > However it has to be done soon or later. We (OpenPower community) are
> > willing to contribute in anyway possible.
> >
> > On Tue, May 31, 2016 at 10:02 PM, Allen Wittenauer <aw...@apache.org>
wrote:
> >
> >>
> >> > On May 31, 2016, at 9:45 AM, MrAsanjar . <af...@gmail.com> wrote:
> >> >
> >> > Thanks Allen for your advise,
> >> > We would have hoped that this problem could have been resolved by
> >> simply getting in touch with the community, the same way as we had done
> >> with Apache Bigtop. Their support has been superb.
> >>
> >> Apache Bigtop is but one project. Adding PPC gear to the ASF
> >> build pool has a significantly larger impact since a lot more of the
Hadoop
> >> ecosystem and ASF project in general are sharing that infrastructure.
> >
> >
> >
Re: Hadoop CI with alternate architectures.
Posted by Upayavira <uv...@odoko.co.uk>.
I would suggest you either mail infrastructure@, or better, create an
INFRA ticket in JIRA. This list isn't used much anymore, and I suspect
isn't where the infrastructure team hang out.
Upayavira
On Wed, 8 Jun 2016, at 01:13 PM, MrAsanjar . wrote:
> Apache Build team, any comments in this regard would be greatly
> appreciated.
> https://issues.apache.org/jira/browse/INFRA-12014
>
> On Tue, May 31, 2016 at 7:28 PM, MrAsanjar . <af...@gmail.com> wrote:
>
> > Allen, I couldn't agree more with the opinions stated in your reply.
> > However it has to be done soon or later. We (OpenPower community) are
> > willing to contribute in anyway possible.
> >
> > On Tue, May 31, 2016 at 10:02 PM, Allen Wittenauer <aw...@apache.org> wrote:
> >
> >>
> >> > On May 31, 2016, at 9:45 AM, MrAsanjar . <af...@gmail.com> wrote:
> >> >
> >> > Thanks Allen for your advise,
> >> > We would have hoped that this problem could have been resolved by
> >> simply getting in touch with the community, the same way as we had done
> >> with Apache Bigtop. Their support has been superb.
> >>
> >> Apache Bigtop is but one project. Adding PPC gear to the ASF
> >> build pool has a significantly larger impact since a lot more of the Hadoop
> >> ecosystem and ASF project in general are sharing that infrastructure.
> >
> >
> >
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
Apache Build team, any comments in this regard would be greatly appreciated.
https://issues.apache.org/jira/browse/INFRA-12014
On Tue, May 31, 2016 at 7:28 PM, MrAsanjar . <af...@gmail.com> wrote:
> Allen, I couldn't agree more with the opinions stated in your reply.
> However it has to be done soon or later. We (OpenPower community) are
> willing to contribute in anyway possible.
>
> On Tue, May 31, 2016 at 10:02 PM, Allen Wittenauer <aw...@apache.org> wrote:
>
>>
>> > On May 31, 2016, at 9:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>> >
>> > Thanks Allen for your advise,
>> > We would have hoped that this problem could have been resolved by
>> simply getting in touch with the community, the same way as we had done
>> with Apache Bigtop. Their support has been superb.
>>
>> Apache Bigtop is but one project. Adding PPC gear to the ASF
>> build pool has a significantly larger impact since a lot more of the Hadoop
>> ecosystem and ASF project in general are sharing that infrastructure.
>
>
>
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
Allen, I couldn't agree more with the opinions stated in your reply.
However it has to be done soon or later. We (OpenPower community) are
willing to contribute in anyway possible.
On Tue, May 31, 2016 at 10:02 PM, Allen Wittenauer <aw...@apache.org> wrote:
>
> > On May 31, 2016, at 9:45 AM, MrAsanjar . <af...@gmail.com> wrote:
> >
> > Thanks Allen for your advise,
> > We would have hoped that this problem could have been resolved by simply
> getting in touch with the community, the same way as we had done with
> Apache Bigtop. Their support has been superb.
>
> Apache Bigtop is but one project. Adding PPC gear to the ASF
> build pool has a significantly larger impact since a lot more of the Hadoop
> ecosystem and ASF project in general are sharing that infrastructure.
Re: Hadoop CI with alternate architectures.
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 31, 2016, at 9:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>
> Thanks Allen for your advise,
> We would have hoped that this problem could have been resolved by simply getting in touch with the community, the same way as we had done with Apache Bigtop. Their support has been superb.
Apache Bigtop is but one project. Adding PPC gear to the ASF build pool has a significantly larger impact since a lot more of the Hadoop ecosystem and ASF project in general are sharing that infrastructure.
Re: Hadoop CI with alternate architectures.
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 31, 2016, at 9:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>
> Thanks Allen for your advise,
> We would have hoped that this problem could have been resolved by simply getting in touch with the community, the same way as we had done with Apache Bigtop. Their support has been superb.
Apache Bigtop is but one project. Adding PPC gear to the ASF build pool has a significantly larger impact since a lot more of the Hadoop ecosystem and ASF project in general are sharing that infrastructure.
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
Thanks Allen for your advise,
We would have hoped that this problem could have been resolved by simply
getting in touch with the community, the same way as we had done with
Apache Bigtop. Their support has been superb.
Thanks
Amir Sanjar
On Tue, May 31, 2016 at 3:33 PM, Allen Wittenauer <aw...@apache.org> wrote:
>
> Since infra-dev appears to be staying out of the conversation, I’d
> recommend filing a JIRA under INFRA to get their attention.
>
> Thanks.
>
> > On May 23, 2016, at 9:57 PM, MrAsanjar . <af...@gmail.com> wrote:
> >
> > back to top:
> > Any suggestion from apache builds team on how to proceed?
> > I have at least one ppc64le build server readied to contribute to the
> build pool
> >
> > On Thu, May 19, 2016 at 10:27 AM, MrAsanjar . <af...@gmail.com>
> wrote:
> > Thanks Steve,
> > Adding builds@apache.org to the list.
> > I do realize the primary concern is the post build, fixing build breaks
> and bugs.
> > Le me assure you that is our primary objective as well, fixing CI issues.
> >
> > On Wed, May 18, 2016 at 12:46 PM, MrAsanjar . <af...@gmail.com>
> wrote:
> > Allen/Tsuyoshi
> > Thank you for your prompt replies. Well, let me fix CI for ppc
> architecture issue once and for all :)
> > We at OpenPOWER foundation have at least one Power system with the
> following configuration available:
> > Power8, 4 cores, 32 G mem, 500 G HD.
> > Would that be sufficient to start?
> >
> > On Wed, May 18, 2016 at 10:29 AM, Allen Wittenauer <aw...@apache.org>
> wrote:
> >
> >
> > That’s really a question for infrastructure-dev@apache.org .
> They manage the ASF build infrastructure which Apache Hadoop and lots of
> other projects utilize. (Bigtop uses something custom, which I think is
> funded by Cloudera.)
> >
> > Once it is registered with builds.apache.org, it’s just a
> matter of us enabling a Jenkins job for it. I suspect we’ll likely do it
> like we have done other architectures in the past: set up a nightly job
> that people will need to cognizant of. Adding it to precommit would likely
> be too much and/or require some significant Yetus work to support reports
> from multiple hosts.
> >
> > FWIW:
> >
> > * We have a Mac mini which, ironically, I thought I’d
> see if I could get a nightly job running on while I type this.
> > * We have a dedicated Windows box which appears to be
> completely screwed up.
> > * In the past, I think it was IBM that hinted that they
> would be willing to give ASF a PowerPC box but that offer seems to have
> disappeared.
> >
> > > On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> > >
> > > Hi Asanjar,
> > >
> > > Thanks for your contribution! I'm ashamed to say, but I don't know how
> > > to change the build machine.
> > >
> > > Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
> > > know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
> > >
> > > Thanks,
> > > - Tsuyoshi
> > >
> > > On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com>
> wrote:
> > >> moving up.. :)
> > >>
> > >> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com>
> wrote:
> > >>
> > >>> I am writing this email to reduce mishaps similar to the issue
> reported by
> > >>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a
> nutshell,
> > >>> an x86 specific
> > >>> performance enhancement broke Hadoop build on Power and SPARC
> architecture.
> > >>> To avoid similar issues in future, could I offer my help here, as a
> > >>> OpenPOWER foundation member.
> > >>> For example, we could contribute a Power based Jenkins slave(s) to
> Apache
> > >>> Hadoop CI. As we have successfully done similar contribution to
> Apache
> > >>> Bigtop CI in past
> > >>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we
> could
> > >>> catch any regressions earlier in Hadoop development cycle. I'd
> appreciate
> > >>> community's guidance on this.
> > >>>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
> >
> >
> >
>
>
Re: Hadoop CI with alternate architectures.
Posted by Allen Wittenauer <aw...@apache.org>.
Since infra-dev appears to be staying out of the conversation, I’d recommend filing a JIRA under INFRA to get their attention.
Thanks.
> On May 23, 2016, at 9:57 PM, MrAsanjar . <af...@gmail.com> wrote:
>
> back to top:
> Any suggestion from apache builds team on how to proceed?
> I have at least one ppc64le build server readied to contribute to the build pool
>
> On Thu, May 19, 2016 at 10:27 AM, MrAsanjar . <af...@gmail.com> wrote:
> Thanks Steve,
> Adding builds@apache.org to the list.
> I do realize the primary concern is the post build, fixing build breaks and bugs.
> Le me assure you that is our primary objective as well, fixing CI issues.
>
> On Wed, May 18, 2016 at 12:46 PM, MrAsanjar . <af...@gmail.com> wrote:
> Allen/Tsuyoshi
> Thank you for your prompt replies. Well, let me fix CI for ppc architecture issue once and for all :)
> We at OpenPOWER foundation have at least one Power system with the following configuration available:
> Power8, 4 cores, 32 G mem, 500 G HD.
> Would that be sufficient to start?
>
> On Wed, May 18, 2016 at 10:29 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
>
> That’s really a question for infrastructure-dev@apache.org . They manage the ASF build infrastructure which Apache Hadoop and lots of other projects utilize. (Bigtop uses something custom, which I think is funded by Cloudera.)
>
> Once it is registered with builds.apache.org, it’s just a matter of us enabling a Jenkins job for it. I suspect we’ll likely do it like we have done other architectures in the past: set up a nightly job that people will need to cognizant of. Adding it to precommit would likely be too much and/or require some significant Yetus work to support reports from multiple hosts.
>
> FWIW:
>
> * We have a Mac mini which, ironically, I thought I’d see if I could get a nightly job running on while I type this.
> * We have a dedicated Windows box which appears to be completely screwed up.
> * In the past, I think it was IBM that hinted that they would be willing to give ASF a PowerPC box but that offer seems to have disappeared.
>
> > On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> > Hi Asanjar,
> >
> > Thanks for your contribution! I'm ashamed to say, but I don't know how
> > to change the build machine.
> >
> > Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
> > know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
> >
> > Thanks,
> > - Tsuyoshi
> >
> > On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com> wrote:
> >> moving up.. :)
> >>
> >> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
> >>
> >>> I am writing this email to reduce mishaps similar to the issue reported by
> >>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
> >>> an x86 specific
> >>> performance enhancement broke Hadoop build on Power and SPARC architecture.
> >>> To avoid similar issues in future, could I offer my help here, as a
> >>> OpenPOWER foundation member.
> >>> For example, we could contribute a Power based Jenkins slave(s) to Apache
> >>> Hadoop CI. As we have successfully done similar contribution to Apache
> >>> Bigtop CI in past
> >>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
> >>> catch any regressions earlier in Hadoop development cycle. I'd appreciate
> >>> community's guidance on this.
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
>
>
>
Re: Hadoop CI with alternate architectures.
Posted by Allen Wittenauer <aw...@apache.org>.
Since infra-dev appears to be staying out of the conversation, I’d recommend filing a JIRA under INFRA to get their attention.
Thanks.
> On May 23, 2016, at 9:57 PM, MrAsanjar . <af...@gmail.com> wrote:
>
> back to top:
> Any suggestion from apache builds team on how to proceed?
> I have at least one ppc64le build server readied to contribute to the build pool
>
> On Thu, May 19, 2016 at 10:27 AM, MrAsanjar . <af...@gmail.com> wrote:
> Thanks Steve,
> Adding builds@apache.org to the list.
> I do realize the primary concern is the post build, fixing build breaks and bugs.
> Le me assure you that is our primary objective as well, fixing CI issues.
>
> On Wed, May 18, 2016 at 12:46 PM, MrAsanjar . <af...@gmail.com> wrote:
> Allen/Tsuyoshi
> Thank you for your prompt replies. Well, let me fix CI for ppc architecture issue once and for all :)
> We at OpenPOWER foundation have at least one Power system with the following configuration available:
> Power8, 4 cores, 32 G mem, 500 G HD.
> Would that be sufficient to start?
>
> On Wed, May 18, 2016 at 10:29 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
>
> That’s really a question for infrastructure-dev@apache.org . They manage the ASF build infrastructure which Apache Hadoop and lots of other projects utilize. (Bigtop uses something custom, which I think is funded by Cloudera.)
>
> Once it is registered with builds.apache.org, it’s just a matter of us enabling a Jenkins job for it. I suspect we’ll likely do it like we have done other architectures in the past: set up a nightly job that people will need to cognizant of. Adding it to precommit would likely be too much and/or require some significant Yetus work to support reports from multiple hosts.
>
> FWIW:
>
> * We have a Mac mini which, ironically, I thought I’d see if I could get a nightly job running on while I type this.
> * We have a dedicated Windows box which appears to be completely screwed up.
> * In the past, I think it was IBM that hinted that they would be willing to give ASF a PowerPC box but that offer seems to have disappeared.
>
> > On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> > Hi Asanjar,
> >
> > Thanks for your contribution! I'm ashamed to say, but I don't know how
> > to change the build machine.
> >
> > Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
> > know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
> >
> > Thanks,
> > - Tsuyoshi
> >
> > On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com> wrote:
> >> moving up.. :)
> >>
> >> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
> >>
> >>> I am writing this email to reduce mishaps similar to the issue reported by
> >>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
> >>> an x86 specific
> >>> performance enhancement broke Hadoop build on Power and SPARC architecture.
> >>> To avoid similar issues in future, could I offer my help here, as a
> >>> OpenPOWER foundation member.
> >>> For example, we could contribute a Power based Jenkins slave(s) to Apache
> >>> Hadoop CI. As we have successfully done similar contribution to Apache
> >>> Bigtop CI in past
> >>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
> >>> catch any regressions earlier in Hadoop development cycle. I'd appreciate
> >>> community's guidance on this.
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
>
>
>
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
back to top:
Any suggestion from apache builds team on how to proceed?
I have at least one ppc64le build server readied to contribute to the build
pool
On Thu, May 19, 2016 at 10:27 AM, MrAsanjar . <af...@gmail.com> wrote:
> Thanks Steve,
> Adding builds@apache.org to the list.
> I do realize the primary concern is the post build, fixing build breaks
> and bugs.
> Le me assure you that is our primary objective as well, fixing CI issues.
>
> On Wed, May 18, 2016 at 12:46 PM, MrAsanjar . <af...@gmail.com> wrote:
>
>> Allen/Tsuyoshi
>> Thank you for your prompt replies. Well, let me fix CI for ppc
>> architecture issue once and for all :)
>> We at OpenPOWER foundation have at least one Power system with the
>> following configuration available:
>> Power8, 4 cores, 32 G mem, 500 G HD.
>> Would that be sufficient to start?
>>
>> On Wed, May 18, 2016 at 10:29 AM, Allen Wittenauer <aw...@apache.org> wrote:
>>
>>>
>>>
>>> That’s really a question for infrastructure-dev@apache.org .
>>> They manage the ASF build infrastructure which Apache Hadoop and lots of
>>> other projects utilize. (Bigtop uses something custom, which I think is
>>> funded by Cloudera.)
>>>
>>> Once it is registered with builds.apache.org, it’s just a
>>> matter of us enabling a Jenkins job for it. I suspect we’ll likely do it
>>> like we have done other architectures in the past: set up a nightly job
>>> that people will need to cognizant of. Adding it to precommit would likely
>>> be too much and/or require some significant Yetus work to support reports
>>> from multiple hosts.
>>>
>>> FWIW:
>>>
>>> * We have a Mac mini which, ironically, I thought I’d
>>> see if I could get a nightly job running on while I type this.
>>> * We have a dedicated Windows box which appears to be
>>> completely screwed up.
>>> * In the past, I think it was IBM that hinted that they
>>> would be willing to give ASF a PowerPC box but that offer seems to have
>>> disappeared.
>>>
>>> > On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>> >
>>> > Hi Asanjar,
>>> >
>>> > Thanks for your contribution! I'm ashamed to say, but I don't know how
>>> > to change the build machine.
>>> >
>>> > Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
>>> > know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
>>> >
>>> > Thanks,
>>> > - Tsuyoshi
>>> >
>>> > On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com>
>>> wrote:
>>> >> moving up.. :)
>>> >>
>>> >> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com>
>>> wrote:
>>> >>
>>> >>> I am writing this email to reduce mishaps similar to the issue
>>> reported by
>>> >>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a
>>> nutshell,
>>> >>> an x86 specific
>>> >>> performance enhancement broke Hadoop build on Power and SPARC
>>> architecture.
>>> >>> To avoid similar issues in future, could I offer my help here, as a
>>> >>> OpenPOWER foundation member.
>>> >>> For example, we could contribute a Power based Jenkins slave(s) to
>>> Apache
>>> >>> Hadoop CI. As we have successfully done similar contribution to
>>> Apache
>>> >>> Bigtop CI in past
>>> >>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we
>>> could
>>> >>> catch any regressions earlier in Hadoop development cycle. I'd
>>> appreciate
>>> >>> community's guidance on this.
>>> >>>
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> >
>>>
>>>
>>
>
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
Thanks Steve,
Adding builds@apache.org to the list.
I do realize the primary concern is the post build, fixing build breaks and
bugs.
Le me assure you that is our primary objective as well, fixing CI issues.
On Wed, May 18, 2016 at 12:46 PM, MrAsanjar . <af...@gmail.com> wrote:
> Allen/Tsuyoshi
> Thank you for your prompt replies. Well, let me fix CI for ppc
> architecture issue once and for all :)
> We at OpenPOWER foundation have at least one Power system with the
> following configuration available:
> Power8, 4 cores, 32 G mem, 500 G HD.
> Would that be sufficient to start?
>
> On Wed, May 18, 2016 at 10:29 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
>>
>>
>> That’s really a question for infrastructure-dev@apache.org .
>> They manage the ASF build infrastructure which Apache Hadoop and lots of
>> other projects utilize. (Bigtop uses something custom, which I think is
>> funded by Cloudera.)
>>
>> Once it is registered with builds.apache.org, it’s just a matter
>> of us enabling a Jenkins job for it. I suspect we’ll likely do it like we
>> have done other architectures in the past: set up a nightly job that people
>> will need to cognizant of. Adding it to precommit would likely be too much
>> and/or require some significant Yetus work to support reports from multiple
>> hosts.
>>
>> FWIW:
>>
>> * We have a Mac mini which, ironically, I thought I’d see
>> if I could get a nightly job running on while I type this.
>> * We have a dedicated Windows box which appears to be
>> completely screwed up.
>> * In the past, I think it was IBM that hinted that they
>> would be willing to give ASF a PowerPC box but that offer seems to have
>> disappeared.
>>
>> > On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> >
>> > Hi Asanjar,
>> >
>> > Thanks for your contribution! I'm ashamed to say, but I don't know how
>> > to change the build machine.
>> >
>> > Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
>> > know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
>> >
>> > Thanks,
>> > - Tsuyoshi
>> >
>> > On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com>
>> wrote:
>> >> moving up.. :)
>> >>
>> >> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com>
>> wrote:
>> >>
>> >>> I am writing this email to reduce mishaps similar to the issue
>> reported by
>> >>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a
>> nutshell,
>> >>> an x86 specific
>> >>> performance enhancement broke Hadoop build on Power and SPARC
>> architecture.
>> >>> To avoid similar issues in future, could I offer my help here, as a
>> >>> OpenPOWER foundation member.
>> >>> For example, we could contribute a Power based Jenkins slave(s) to
>> Apache
>> >>> Hadoop CI. As we have successfully done similar contribution to
>> Apache
>> >>> Bigtop CI in past
>> >>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we
>> could
>> >>> catch any regressions earlier in Hadoop development cycle. I'd
>> appreciate
>> >>> community's guidance on this.
>> >>>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >
>>
>>
>
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
Allen/Tsuyoshi
Thank you for your prompt replies. Well, let me fix CI for ppc architecture
issue once and for all :)
We at OpenPOWER foundation have at least one Power system with the
following configuration available:
Power8, 4 cores, 32 G mem, 500 G HD.
Would that be sufficient to start?
On Wed, May 18, 2016 at 10:29 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
>
> That’s really a question for infrastructure-dev@apache.org . They
> manage the ASF build infrastructure which Apache Hadoop and lots of other
> projects utilize. (Bigtop uses something custom, which I think is funded
> by Cloudera.)
>
> Once it is registered with builds.apache.org, it’s just a matter
> of us enabling a Jenkins job for it. I suspect we’ll likely do it like we
> have done other architectures in the past: set up a nightly job that people
> will need to cognizant of. Adding it to precommit would likely be too much
> and/or require some significant Yetus work to support reports from multiple
> hosts.
>
> FWIW:
>
> * We have a Mac mini which, ironically, I thought I’d see
> if I could get a nightly job running on while I type this.
> * We have a dedicated Windows box which appears to be
> completely screwed up.
> * In the past, I think it was IBM that hinted that they
> would be willing to give ASF a PowerPC box but that offer seems to have
> disappeared.
>
> > On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> > Hi Asanjar,
> >
> > Thanks for your contribution! I'm ashamed to say, but I don't know how
> > to change the build machine.
> >
> > Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
> > know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
> >
> > Thanks,
> > - Tsuyoshi
> >
> > On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com>
> wrote:
> >> moving up.. :)
> >>
> >> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com>
> wrote:
> >>
> >>> I am writing this email to reduce mishaps similar to the issue
> reported by
> >>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a
> nutshell,
> >>> an x86 specific
> >>> performance enhancement broke Hadoop build on Power and SPARC
> architecture.
> >>> To avoid similar issues in future, could I offer my help here, as a
> >>> OpenPOWER foundation member.
> >>> For example, we could contribute a Power based Jenkins slave(s) to
> Apache
> >>> Hadoop CI. As we have successfully done similar contribution to Apache
> >>> Bigtop CI in past
> >>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we
> could
> >>> catch any regressions earlier in Hadoop development cycle. I'd
> appreciate
> >>> community's guidance on this.
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
>
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
Allen/Tsuyoshi
Thank you for your prompt replies. Well, let me fix CI for ppc architecture
issue once and for all :)
We at OpenPOWER foundation have at least one Power system with the
following configuration available:
Power8, 4 cores, 32 G mem, 500 G HD.
Would that be sufficient to start?
On Wed, May 18, 2016 at 10:29 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
>
> That’s really a question for infrastructure-dev@apache.org . They
> manage the ASF build infrastructure which Apache Hadoop and lots of other
> projects utilize. (Bigtop uses something custom, which I think is funded
> by Cloudera.)
>
> Once it is registered with builds.apache.org, it’s just a matter
> of us enabling a Jenkins job for it. I suspect we’ll likely do it like we
> have done other architectures in the past: set up a nightly job that people
> will need to cognizant of. Adding it to precommit would likely be too much
> and/or require some significant Yetus work to support reports from multiple
> hosts.
>
> FWIW:
>
> * We have a Mac mini which, ironically, I thought I’d see
> if I could get a nightly job running on while I type this.
> * We have a dedicated Windows box which appears to be
> completely screwed up.
> * In the past, I think it was IBM that hinted that they
> would be willing to give ASF a PowerPC box but that offer seems to have
> disappeared.
>
> > On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> > Hi Asanjar,
> >
> > Thanks for your contribution! I'm ashamed to say, but I don't know how
> > to change the build machine.
> >
> > Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
> > know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
> >
> > Thanks,
> > - Tsuyoshi
> >
> > On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com>
> wrote:
> >> moving up.. :)
> >>
> >> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com>
> wrote:
> >>
> >>> I am writing this email to reduce mishaps similar to the issue
> reported by
> >>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a
> nutshell,
> >>> an x86 specific
> >>> performance enhancement broke Hadoop build on Power and SPARC
> architecture.
> >>> To avoid similar issues in future, could I offer my help here, as a
> >>> OpenPOWER foundation member.
> >>> For example, we could contribute a Power based Jenkins slave(s) to
> Apache
> >>> Hadoop CI. As we have successfully done similar contribution to Apache
> >>> Bigtop CI in past
> >>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we
> could
> >>> catch any regressions earlier in Hadoop development cycle. I'd
> appreciate
> >>> community's guidance on this.
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
>
Re: Hadoop CI with alternate architectures.
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, May 18, 2016 at 08:29AM, Allen Wittenauer wrote:
> That\u2019s really a question for infrastructure-dev@apache.org . They
> manage the ASF build infrastructure which Apache Hadoop and lots of
> other projects utilize. (Bigtop uses something custom, which I think
> is funded by Cloudera.)
Nope, Bigtop CI is not sponsored by Cloudera.
Besides, Bigtop is running itw own CI because we are producing a lot of
binary artifacts and don't won't to run a risk of clogging ASF-wide CI.
Cos
--
Take care,
Konstantin (Cos) Boudnik
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Steve Loughran <st...@hortonworks.com>.
> On 18 May 2016, at 16:29, Allen Wittenauer <aw...@apache.org> wrote:
>
>
>
> That’s really a question for infrastructure-dev@apache.org . They manage the ASF build infrastructure which Apache Hadoop and lots of other projects utilize. (Bigtop uses something custom, which I think is funded by Cloudera.)
There's no dedicated Hadoop boxes, just a pool of machines for all projects. I don't know about bigtop.
There's a mailing list, builds@apache.org, for people who want to get involved
>
> Once it is registered with builds.apache.org, it’s just a matter of us enabling a Jenkins job for it. I suspect we’ll likely do it like we have done other architectures in the past: set up a nightly job that people will need to cognizant of. Adding it to precommit would likely be too much and/or require some significant Yetus work to support reports from multiple hosts.
>
Nightly would be a good initial step; the hard part is getting it stable enough for its build-failed messages to not get ignored (it should be on send-no-emails mode until stable). Then comes the cultural shift: getting people to care that the build is failing. That'll be from native, OS and JVM issues.
Maybe Yetus could be set to warn that on changes which go near the .C or makefile code, that someone should explicitly kick off a precommit run on the PPC. That could be enough to remind those people playing with native code that that's something that really needs a check
> FWIW:
>
> * We have a Mac mini which, ironically, I thought I’d see if I could get a nightly job running on while I type this.
> * We have a dedicated Windows box which appears to be completely screwed up.
the default state of all windows boxes, sadly. The VMs are odd too; the FS permissions are enough to stop MiniYarnClusters coming up.
> * In the past, I think it was IBM that hinted that they would be willing to give ASF a PowerPC box but that offer seems to have disappeared.
>
Re: Hadoop CI with alternate architectures.
Posted by Steve Loughran <st...@hortonworks.com>.
> On 18 May 2016, at 16:29, Allen Wittenauer <aw...@apache.org> wrote:
>
>
>
> That’s really a question for infrastructure-dev@apache.org . They manage the ASF build infrastructure which Apache Hadoop and lots of other projects utilize. (Bigtop uses something custom, which I think is funded by Cloudera.)
There's no dedicated Hadoop boxes, just a pool of machines for all projects. I don't know about bigtop.
There's a mailing list, builds@apache.org, for people who want to get involved
>
> Once it is registered with builds.apache.org, it’s just a matter of us enabling a Jenkins job for it. I suspect we’ll likely do it like we have done other architectures in the past: set up a nightly job that people will need to cognizant of. Adding it to precommit would likely be too much and/or require some significant Yetus work to support reports from multiple hosts.
>
Nightly would be a good initial step; the hard part is getting it stable enough for its build-failed messages to not get ignored (it should be on send-no-emails mode until stable). Then comes the cultural shift: getting people to care that the build is failing. That'll be from native, OS and JVM issues.
Maybe Yetus could be set to warn that on changes which go near the .C or makefile code, that someone should explicitly kick off a precommit run on the PPC. That could be enough to remind those people playing with native code that that's something that really needs a check
> FWIW:
>
> * We have a Mac mini which, ironically, I thought I’d see if I could get a nightly job running on while I type this.
> * We have a dedicated Windows box which appears to be completely screwed up.
the default state of all windows boxes, sadly. The VMs are odd too; the FS permissions are enough to stop MiniYarnClusters coming up.
> * In the past, I think it was IBM that hinted that they would be willing to give ASF a PowerPC box but that offer seems to have disappeared.
>
Re: Hadoop CI with alternate architectures.
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, May 18, 2016 at 08:29AM, Allen Wittenauer wrote:
> That\u2019s really a question for infrastructure-dev@apache.org . They
> manage the ASF build infrastructure which Apache Hadoop and lots of
> other projects utilize. (Bigtop uses something custom, which I think
> is funded by Cloudera.)
Nope, Bigtop CI is not sponsored by Cloudera.
Besides, Bigtop is running itw own CI because we are producing a lot of
binary artifacts and don't won't to run a risk of clogging ASF-wide CI.
Cos
--
Take care,
Konstantin (Cos) Boudnik
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, May 18, 2016 at 08:29AM, Allen Wittenauer wrote:
> That\u2019s really a question for infrastructure-dev@apache.org . They
> manage the ASF build infrastructure which Apache Hadoop and lots of
> other projects utilize. (Bigtop uses something custom, which I think
> is funded by Cloudera.)
Nope, Bigtop CI is not sponsored by Cloudera.
Besides, Bigtop is running itw own CI because we are producing a lot of
binary artifacts and don't won't to run a risk of clogging ASF-wide CI.
Cos
--
Take care,
Konstantin (Cos) Boudnik
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, May 18, 2016 at 08:29AM, Allen Wittenauer wrote:
> That\u2019s really a question for infrastructure-dev@apache.org . They
> manage the ASF build infrastructure which Apache Hadoop and lots of
> other projects utilize. (Bigtop uses something custom, which I think
> is funded by Cloudera.)
Nope, Bigtop CI is not sponsored by Cloudera.
Besides, Bigtop is running itw own CI because we are producing a lot of
binary artifacts and don't won't to run a risk of clogging ASF-wide CI.
Cos
--
Take care,
Konstantin (Cos) Boudnik
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Steve Loughran <st...@hortonworks.com>.
> On 18 May 2016, at 16:29, Allen Wittenauer <aw...@apache.org> wrote:
>
>
>
> That’s really a question for infrastructure-dev@apache.org . They manage the ASF build infrastructure which Apache Hadoop and lots of other projects utilize. (Bigtop uses something custom, which I think is funded by Cloudera.)
There's no dedicated Hadoop boxes, just a pool of machines for all projects. I don't know about bigtop.
There's a mailing list, builds@apache.org, for people who want to get involved
>
> Once it is registered with builds.apache.org, it’s just a matter of us enabling a Jenkins job for it. I suspect we’ll likely do it like we have done other architectures in the past: set up a nightly job that people will need to cognizant of. Adding it to precommit would likely be too much and/or require some significant Yetus work to support reports from multiple hosts.
>
Nightly would be a good initial step; the hard part is getting it stable enough for its build-failed messages to not get ignored (it should be on send-no-emails mode until stable). Then comes the cultural shift: getting people to care that the build is failing. That'll be from native, OS and JVM issues.
Maybe Yetus could be set to warn that on changes which go near the .C or makefile code, that someone should explicitly kick off a precommit run on the PPC. That could be enough to remind those people playing with native code that that's something that really needs a check
> FWIW:
>
> * We have a Mac mini which, ironically, I thought I’d see if I could get a nightly job running on while I type this.
> * We have a dedicated Windows box which appears to be completely screwed up.
the default state of all windows boxes, sadly. The VMs are odd too; the FS permissions are enough to stop MiniYarnClusters coming up.
> * In the past, I think it was IBM that hinted that they would be willing to give ASF a PowerPC box but that offer seems to have disappeared.
>
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
Allen/Tsuyoshi
Thank you for your prompt replies. Well, let me fix CI for ppc architecture
issue once and for all :)
We at OpenPOWER foundation have at least one Power system with the
following configuration available:
Power8, 4 cores, 32 G mem, 500 G HD.
Would that be sufficient to start?
On Wed, May 18, 2016 at 10:29 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
>
> That’s really a question for infrastructure-dev@apache.org . They
> manage the ASF build infrastructure which Apache Hadoop and lots of other
> projects utilize. (Bigtop uses something custom, which I think is funded
> by Cloudera.)
>
> Once it is registered with builds.apache.org, it’s just a matter
> of us enabling a Jenkins job for it. I suspect we’ll likely do it like we
> have done other architectures in the past: set up a nightly job that people
> will need to cognizant of. Adding it to precommit would likely be too much
> and/or require some significant Yetus work to support reports from multiple
> hosts.
>
> FWIW:
>
> * We have a Mac mini which, ironically, I thought I’d see
> if I could get a nightly job running on while I type this.
> * We have a dedicated Windows box which appears to be
> completely screwed up.
> * In the past, I think it was IBM that hinted that they
> would be willing to give ASF a PowerPC box but that offer seems to have
> disappeared.
>
> > On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> > Hi Asanjar,
> >
> > Thanks for your contribution! I'm ashamed to say, but I don't know how
> > to change the build machine.
> >
> > Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
> > know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
> >
> > Thanks,
> > - Tsuyoshi
> >
> > On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com>
> wrote:
> >> moving up.. :)
> >>
> >> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com>
> wrote:
> >>
> >>> I am writing this email to reduce mishaps similar to the issue
> reported by
> >>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a
> nutshell,
> >>> an x86 specific
> >>> performance enhancement broke Hadoop build on Power and SPARC
> architecture.
> >>> To avoid similar issues in future, could I offer my help here, as a
> >>> OpenPOWER foundation member.
> >>> For example, we could contribute a Power based Jenkins slave(s) to
> Apache
> >>> Hadoop CI. As we have successfully done similar contribution to Apache
> >>> Bigtop CI in past
> >>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we
> could
> >>> catch any regressions earlier in Hadoop development cycle. I'd
> appreciate
> >>> community's guidance on this.
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
>
Re: Hadoop CI with alternate architectures.
Posted by Allen Wittenauer <aw...@apache.org>.
That’s really a question for infrastructure-dev@apache.org . They manage the ASF build infrastructure which Apache Hadoop and lots of other projects utilize. (Bigtop uses something custom, which I think is funded by Cloudera.)
Once it is registered with builds.apache.org, it’s just a matter of us enabling a Jenkins job for it. I suspect we’ll likely do it like we have done other architectures in the past: set up a nightly job that people will need to cognizant of. Adding it to precommit would likely be too much and/or require some significant Yetus work to support reports from multiple hosts.
FWIW:
* We have a Mac mini which, ironically, I thought I’d see if I could get a nightly job running on while I type this.
* We have a dedicated Windows box which appears to be completely screwed up.
* In the past, I think it was IBM that hinted that they would be willing to give ASF a PowerPC box but that offer seems to have disappeared.
> On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> Hi Asanjar,
>
> Thanks for your contribution! I'm ashamed to say, but I don't know how
> to change the build machine.
>
> Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
> know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
>
> Thanks,
> - Tsuyoshi
>
> On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com> wrote:
>> moving up.. :)
>>
>> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>>
>>> I am writing this email to reduce mishaps similar to the issue reported by
>>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
>>> an x86 specific
>>> performance enhancement broke Hadoop build on Power and SPARC architecture.
>>> To avoid similar issues in future, could I offer my help here, as a
>>> OpenPOWER foundation member.
>>> For example, we could contribute a Power based Jenkins slave(s) to Apache
>>> Hadoop CI. As we have successfully done similar contribution to Apache
>>> Bigtop CI in past
>>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
>>> catch any regressions earlier in Hadoop development cycle. I'd appreciate
>>> community's guidance on this.
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Allen Wittenauer <aw...@apache.org>.
That’s really a question for infrastructure-dev@apache.org . They manage the ASF build infrastructure which Apache Hadoop and lots of other projects utilize. (Bigtop uses something custom, which I think is funded by Cloudera.)
Once it is registered with builds.apache.org, it’s just a matter of us enabling a Jenkins job for it. I suspect we’ll likely do it like we have done other architectures in the past: set up a nightly job that people will need to cognizant of. Adding it to precommit would likely be too much and/or require some significant Yetus work to support reports from multiple hosts.
FWIW:
* We have a Mac mini which, ironically, I thought I’d see if I could get a nightly job running on while I type this.
* We have a dedicated Windows box which appears to be completely screwed up.
* In the past, I think it was IBM that hinted that they would be willing to give ASF a PowerPC box but that offer seems to have disappeared.
> On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> Hi Asanjar,
>
> Thanks for your contribution! I'm ashamed to say, but I don't know how
> to change the build machine.
>
> Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
> know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
>
> Thanks,
> - Tsuyoshi
>
> On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com> wrote:
>> moving up.. :)
>>
>> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>>
>>> I am writing this email to reduce mishaps similar to the issue reported by
>>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
>>> an x86 specific
>>> performance enhancement broke Hadoop build on Power and SPARC architecture.
>>> To avoid similar issues in future, could I offer my help here, as a
>>> OpenPOWER foundation member.
>>> For example, we could contribute a Power based Jenkins slave(s) to Apache
>>> Hadoop CI. As we have successfully done similar contribution to Apache
>>> Bigtop CI in past
>>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
>>> catch any regressions earlier in Hadoop development cycle. I'd appreciate
>>> community's guidance on this.
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Allen Wittenauer <aw...@apache.org>.
That’s really a question for infrastructure-dev@apache.org . They manage the ASF build infrastructure which Apache Hadoop and lots of other projects utilize. (Bigtop uses something custom, which I think is funded by Cloudera.)
Once it is registered with builds.apache.org, it’s just a matter of us enabling a Jenkins job for it. I suspect we’ll likely do it like we have done other architectures in the past: set up a nightly job that people will need to cognizant of. Adding it to precommit would likely be too much and/or require some significant Yetus work to support reports from multiple hosts.
FWIW:
* We have a Mac mini which, ironically, I thought I’d see if I could get a nightly job running on while I type this.
* We have a dedicated Windows box which appears to be completely screwed up.
* In the past, I think it was IBM that hinted that they would be willing to give ASF a PowerPC box but that offer seems to have disappeared.
> On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> Hi Asanjar,
>
> Thanks for your contribution! I'm ashamed to say, but I don't know how
> to change the build machine.
>
> Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
> know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
>
> Thanks,
> - Tsuyoshi
>
> On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com> wrote:
>> moving up.. :)
>>
>> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>>
>>> I am writing this email to reduce mishaps similar to the issue reported by
>>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
>>> an x86 specific
>>> performance enhancement broke Hadoop build on Power and SPARC architecture.
>>> To avoid similar issues in future, could I offer my help here, as a
>>> OpenPOWER foundation member.
>>> For example, we could contribute a Power based Jenkins slave(s) to Apache
>>> Hadoop CI. As we have successfully done similar contribution to Apache
>>> Bigtop CI in past
>>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
>>> catch any regressions earlier in Hadoop development cycle. I'd appreciate
>>> community's guidance on this.
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Allen Wittenauer <aw...@apache.org>.
That’s really a question for infrastructure-dev@apache.org . They manage the ASF build infrastructure which Apache Hadoop and lots of other projects utilize. (Bigtop uses something custom, which I think is funded by Cloudera.)
Once it is registered with builds.apache.org, it’s just a matter of us enabling a Jenkins job for it. I suspect we’ll likely do it like we have done other architectures in the past: set up a nightly job that people will need to cognizant of. Adding it to precommit would likely be too much and/or require some significant Yetus work to support reports from multiple hosts.
FWIW:
* We have a Mac mini which, ironically, I thought I’d see if I could get a nightly job running on while I type this.
* We have a dedicated Windows box which appears to be completely screwed up.
* In the past, I think it was IBM that hinted that they would be willing to give ASF a PowerPC box but that offer seems to have disappeared.
> On May 18, 2016, at 7:52 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> Hi Asanjar,
>
> Thanks for your contribution! I'm ashamed to say, but I don't know how
> to change the build machine.
>
> Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
> know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
>
> Thanks,
> - Tsuyoshi
>
> On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com> wrote:
>> moving up.. :)
>>
>> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>>
>>> I am writing this email to reduce mishaps similar to the issue reported by
>>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
>>> an x86 specific
>>> performance enhancement broke Hadoop build on Power and SPARC architecture.
>>> To avoid similar issues in future, could I offer my help here, as a
>>> OpenPOWER foundation member.
>>> For example, we could contribute a Power based Jenkins slave(s) to Apache
>>> Hadoop CI. As we have successfully done similar contribution to Apache
>>> Bigtop CI in past
>>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
>>> catch any regressions earlier in Hadoop development cycle. I'd appreciate
>>> community's guidance on this.
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Hi Asanjar,
Thanks for your contribution! I'm ashamed to say, but I don't know how
to change the build machine.
Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
Thanks,
- Tsuyoshi
On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com> wrote:
> moving up.. :)
>
> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>
>> I am writing this email to reduce mishaps similar to the issue reported by
>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
>> an x86 specific
>> performance enhancement broke Hadoop build on Power and SPARC architecture.
>> To avoid similar issues in future, could I offer my help here, as a
>> OpenPOWER foundation member.
>> For example, we could contribute a Power based Jenkins slave(s) to Apache
>> Hadoop CI. As we have successfully done similar contribution to Apache
>> Bigtop CI in past
>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
>> catch any regressions earlier in Hadoop development cycle. I'd appreciate
>> community's guidance on this.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Hi Asanjar,
Thanks for your contribution! I'm ashamed to say, but I don't know how
to change the build machine.
Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
Thanks,
- Tsuyoshi
On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com> wrote:
> moving up.. :)
>
> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>
>> I am writing this email to reduce mishaps similar to the issue reported by
>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
>> an x86 specific
>> performance enhancement broke Hadoop build on Power and SPARC architecture.
>> To avoid similar issues in future, could I offer my help here, as a
>> OpenPOWER foundation member.
>> For example, we could contribute a Power based Jenkins slave(s) to Apache
>> Hadoop CI. As we have successfully done similar contribution to Apache
>> Bigtop CI in past
>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
>> catch any regressions earlier in Hadoop development cycle. I'd appreciate
>> community's guidance on this.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Hi Asanjar,
Thanks for your contribution! I'm ashamed to say, but I don't know how
to change the build machine.
Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
Thanks,
- Tsuyoshi
On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com> wrote:
> moving up.. :)
>
> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>
>> I am writing this email to reduce mishaps similar to the issue reported by
>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
>> an x86 specific
>> performance enhancement broke Hadoop build on Power and SPARC architecture.
>> To avoid similar issues in future, could I offer my help here, as a
>> OpenPOWER foundation member.
>> For example, we could contribute a Power based Jenkins slave(s) to Apache
>> Hadoop CI. As we have successfully done similar contribution to Apache
>> Bigtop CI in past
>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
>> catch any regressions earlier in Hadoop development cycle. I'd appreciate
>> community's guidance on this.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Hi Asanjar,
Thanks for your contribution! I'm ashamed to say, but I don't know how
to change the build machine.
Adding "to" Allen, who is a PMC of Yetus project. Hey Allen, do you
know how to add Power based Jenkins slave(s) to Apache Hadoop CI?
Thanks,
- Tsuyoshi
On Wed, May 18, 2016 at 11:18 PM, MrAsanjar . <af...@gmail.com> wrote:
> moving up.. :)
>
> On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
>
>> I am writing this email to reduce mishaps similar to the issue reported by
>> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
>> an x86 specific
>> performance enhancement broke Hadoop build on Power and SPARC architecture.
>> To avoid similar issues in future, could I offer my help here, as a
>> OpenPOWER foundation member.
>> For example, we could contribute a Power based Jenkins slave(s) to Apache
>> Hadoop CI. As we have successfully done similar contribution to Apache
>> Bigtop CI in past
>> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
>> catch any regressions earlier in Hadoop development cycle. I'd appreciate
>> community's guidance on this.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
moving up.. :)
On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
> I am writing this email to reduce mishaps similar to the issue reported by
> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
> an x86 specific
> performance enhancement broke Hadoop build on Power and SPARC architecture.
> To avoid similar issues in future, could I offer my help here, as a
> OpenPOWER foundation member.
> For example, we could contribute a Power based Jenkins slave(s) to Apache
> Hadoop CI. As we have successfully done similar contribution to Apache
> Bigtop CI in past
> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
> catch any regressions earlier in Hadoop development cycle. I'd appreciate
> community's guidance on this.
>
Re: Hadoop CI with alternate architectures.
Posted by "MrAsanjar ." <af...@gmail.com>.
moving up.. :)
On Thu, May 12, 2016 at 12:45 AM, MrAsanjar . <af...@gmail.com> wrote:
> I am writing this email to reduce mishaps similar to the issue reported by
> JIRA https://issues.apache.org/jira/browse/HADOOP-11505. In a nutshell,
> an x86 specific
> performance enhancement broke Hadoop build on Power and SPARC architecture.
> To avoid similar issues in future, could I offer my help here, as a
> OpenPOWER foundation member.
> For example, we could contribute a Power based Jenkins slave(s) to Apache
> Hadoop CI. As we have successfully done similar contribution to Apache
> Bigtop CI in past
> https://ci.bigtop.apache.org/computer/docker-slave-ppc-1/. Thus, we could
> catch any regressions earlier in Hadoop development cycle. I'd appreciate
> community's guidance on this.
>