You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Brahma Reddy Battula <br...@huawei.com> on 2016/10/28 03:20:07 UTC
[DISCUSS] Pre-commit build in Windows platform
Hi All
As we supporting the Hadoop in windows, I feel, we should have pre-commit build in windows( atleast in qbt).
Background:
As of now pre-commit will not run on windows, we may end up with following
1) Test cases can fail in windows
2) Compilation Error (now it's failing, after HADOOP-10075)
3) Some feature might not work
It will be good, if we catch all these at time of commit. (Prevention is better than cure..:) )
If it's already discussed and concluded, Please ignore me and let us know the conclusion.
--Brahma Reddy Battula
Re: [DISCUSS] Pre-commit build in Windows platform
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Nov 1, 2016, at 4:00 AM, Brahma Reddy Battula <br...@huawei.com> wrote:
> Thanks for information. Seems to be challenge to get it done. Can we try to get dedicated machine for this..?
We have one. What we don't have is someone dedicated enough to keep it running. It is not a "one and done". Folks will add new or change dependencies and not bother to update or even to tell anyone to update the build servers. Even though the Linux/x86 builds have the build setup as part of the source tree (dev-support/docker/Dockerfile), people *still* ignore it then complain that their new feature isn't getting tested.
FWIW, there are still bits broken even on Linux/x86. MAPREDUCE-6743 is a great view of the apathetic nature of the committer pool that anyone who takes this on will have to work around. Those native MR tasks tests have been known to be broken since qbt was introduced several months ago. (They weren't testing in the nightly builds before, so have likely been broken for a very long time. qbt just made them visible.) Frankly, Windows builds have zero chance if we can't even get the Linux/x86 build stable.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
RE: [DISCUSS] Pre-commit build in Windows platform
Posted by Brahma Reddy Battula <br...@huawei.com>.
Thanks a lot Steve Loughran , Allen Wittenauer ,Ravi Prakash and Mingliang Liu.
[ Steve Loughran Wrote ]
I think a first step would be to have scheduled builds on windows; ideally with people caring that they are playing up.
Anyway,
+1 for more windows testing
[ Brahma Reddy ]
May be we add in qbt it self..? Or separate night pre-commit build..?
[ Allen Wittenauer Wrote ]
I actually tried to get Apache Yetus testing Apache Hadoop on the hadoop-win box last year. (This was before qbt mode existed.) I gave up because the components needed to build trunk weren't installed and it looked like the box itself was ill. I moved on to the Mac build which, while tricky, got it working with a bit of magic. Then they took the Mac away. IBM provided access to a PowerPC machine, so I moved onto the ppc64le build. After a lot of hand wringing, we got it up and running and stable enough to show how broken the build is by our usage of leveldbjni.
It should be noted that getting infra support for non-Linux/x86 builds is a pretty major exercise in frustration. They don't really have the time, those boxes tend to go down often, and rarely get updated. (The Solaris box is running a really old build of Solaris 10. So old, that when it came out, I was still employed by Sun...) If folks are actually serious about adding more platforms, then I'd suggest that some of these big vendors actually cough up some cash to move the precommit infrastructure off of the ASF and onto something more reliable with more platform diversity.
[ Brahma Reddy ]
Thanks for information. Seems to be challenge to get it done. Can we try to get dedicated machine for this..?
[ Mingliang Liu wrote ]
I also like the idea of having a precommit build in Windows. If only the pre-commit infrastructure was reliable. If this is a pain, I see little value making Windows unit test failures release blockers.
[ Brahma Reddy ]
Yes, I think we can do like this, if nobody object on this.
--Brahma Reddy Battula
-----Original Message-----
From: Mingliang Liu [mailto:liuml07@gmail.com]
Sent: 31 October 2016 03:49
To: common-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Pre-commit build in Windows platform
I also like the idea of having a precommit build in Windows. If only the pre-commit infrastructure was reliable. If this is a pain, I see little value making Windows unit test failures release blockers.
Thanks,
Sent from my iPhone
> On Oct 28, 2016, at 8:16 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Oct 27, 2016, at 8:20 PM, Brahma Reddy Battula <br...@huawei.com> wrote:
>>
>> As we supporting the Hadoop in windows, I feel, we should have pre-commit build in windows( atleast in qbt).
>
>
> I actually tried to get Apache Yetus testing Apache Hadoop on the hadoop-win box last year. (This was before qbt mode existed.) I gave up because the components needed to build trunk weren't installed and it looked like the box itself was ill. I moved on to the Mac build which, while tricky, got it working with a bit of magic. Then they took the Mac away. IBM provided access to a PowerPC machine, so I moved onto the ppc64le build. After a lot of hand wringing, we got it up and running and stable enough to show how broken the build is by our usage of leveldbjni.
>
> It should be noted that getting infra support for non-Linux/x86 builds is a pretty major exercise in frustration. They don't really have the time, those boxes tend to go down often, and rarely get updated. (The Solaris box is running a really old build of Solaris 10. So old, that when it came out, I was still employed by Sun...) If folks are actually serious about adding more platforms, then I'd suggest that some of these big vendors actually cough up some cash to move the precommit infrastructure off of the ASF and onto something more reliable with more platform diversity.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Pre-commit build in Windows platform
Posted by Mingliang Liu <li...@gmail.com>.
I also like the idea of having a precommit build in Windows. If only the pre-commit infrastructure was reliable. If this is a pain, I see little value making Windows unit test failures release blockers.
Thanks,
Sent from my iPhone
> On Oct 28, 2016, at 8:16 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Oct 27, 2016, at 8:20 PM, Brahma Reddy Battula <br...@huawei.com> wrote:
>>
>> As we supporting the Hadoop in windows, I feel, we should have pre-commit build in windows( atleast in qbt).
>
>
> I actually tried to get Apache Yetus testing Apache Hadoop on the hadoop-win box last year. (This was before qbt mode existed.) I gave up because the components needed to build trunk weren't installed and it looked like the box itself was ill. I moved on to the Mac build which, while tricky, got it working with a bit of magic. Then they took the Mac away. IBM provided access to a PowerPC machine, so I moved onto the ppc64le build. After a lot of hand wringing, we got it up and running and stable enough to show how broken the build is by our usage of leveldbjni.
>
> It should be noted that getting infra support for non-Linux/x86 builds is a pretty major exercise in frustration. They don't really have the time, those boxes tend to go down often, and rarely get updated. (The Solaris box is running a really old build of Solaris 10. So old, that when it came out, I was still employed by Sun...) If folks are actually serious about adding more platforms, then I'd suggest that some of these big vendors actually cough up some cash to move the precommit infrastructure off of the ASF and onto something more reliable with more platform diversity.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Pre-commit build in Windows platform
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Oct 27, 2016, at 8:20 PM, Brahma Reddy Battula <br...@huawei.com> wrote:
>
> As we supporting the Hadoop in windows, I feel, we should have pre-commit build in windows( atleast in qbt).
I actually tried to get Apache Yetus testing Apache Hadoop on the hadoop-win box last year. (This was before qbt mode existed.) I gave up because the components needed to build trunk weren't installed and it looked like the box itself was ill. I moved on to the Mac build which, while tricky, got it working with a bit of magic. Then they took the Mac away. IBM provided access to a PowerPC machine, so I moved onto the ppc64le build. After a lot of hand wringing, we got it up and running and stable enough to show how broken the build is by our usage of leveldbjni.
It should be noted that getting infra support for non-Linux/x86 builds is a pretty major exercise in frustration. They don't really have the time, those boxes tend to go down often, and rarely get updated. (The Solaris box is running a really old build of Solaris 10. So old, that when it came out, I was still employed by Sun...) If folks are actually serious about adding more platforms, then I'd suggest that some of these big vendors actually cough up some cash to move the precommit infrastructure off of the ASF and onto something more reliable with more platform diversity.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Pre-commit build in Windows platform
Posted by Ravi Prakash <ra...@gmail.com>.
Was anybody ever successful in getting Windows(tm) installed in a VM from
the developer accounts that Microsoft affords Apache?
On Fri, Oct 28, 2016 at 4:44 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> On 28 Oct 2016, at 04:20, Brahma Reddy Battula <
> brahmareddy.battula@huawei.com<ma...@huawei.com>>
> wrote:
>
> Hi All
>
> As we supporting the Hadoop in windows, I feel, we should have pre-commit
> build in windows( atleast in qbt).
>
> We've never had it work too well with Windows precommit & Jenkins; maybe
> someone could volunteer to try again. What's important is not to overwhelm
> the JIRAs with noise that gets ignored.
>
>
>
> Background:
>
> As of now pre-commit will not run on windows, we may end up with following
>
>
> 1) Test cases can fail in windows
>
> 2) Compilation Error (now it's failing, after HADOOP-10075)
>
>
> I've not had branch-3 building for a while, though maybe it's my Windows
> Server VM that's a mess. Maybe I should rm it and grab the limited-lifespan
> dev box, then add the extra Hadoop bits https://developer.microsoft.
> com/en-us/windows/downloads/virtual-machines
>
> 3) Some feature might not work
>
> It will be good, if we catch all these at time of commit. (Prevention is
> better than cure..:) )
>
> I think a first step would be to have scheduled builds on windows; ideally
> with people caring that they are playing up.
>
> Anyway,
>
> +1 for more windows testing
>
Re: [DISCUSS] Pre-commit build in Windows platform
Posted by Steve Loughran <st...@hortonworks.com>.
On 28 Oct 2016, at 04:20, Brahma Reddy Battula <br...@huawei.com>> wrote:
Hi All
As we supporting the Hadoop in windows, I feel, we should have pre-commit build in windows( atleast in qbt).
We've never had it work too well with Windows precommit & Jenkins; maybe someone could volunteer to try again. What's important is not to overwhelm the JIRAs with noise that gets ignored.
Background:
As of now pre-commit will not run on windows, we may end up with following
1) Test cases can fail in windows
2) Compilation Error (now it's failing, after HADOOP-10075)
I've not had branch-3 building for a while, though maybe it's my Windows Server VM that's a mess. Maybe I should rm it and grab the limited-lifespan dev box, then add the extra Hadoop bits https://developer.microsoft.com/en-us/windows/downloads/virtual-machines
3) Some feature might not work
It will be good, if we catch all these at time of commit. (Prevention is better than cure..:) )
I think a first step would be to have scheduled builds on windows; ideally with people caring that they are playing up.
Anyway,
+1 for more windows testing