Re: YARN precommit busted?

[Ted Yu <yu...@gmail.com>]

With continuous failed QA runs on H5 and H6, I have to take them out again.

The INFRA has bounced these two machines. Unfortunately that didn't solve
the issue.

On Fri, Dec 30, 2016 at 2:42 PM, Ted Yu <yu...@gmail.com> wrote:

> Put back H5 and H6 since the INFRA ticket has been resolved.
> Excluded H9.
>
> On Fri, Dec 30, 2016 at 2:36 PM, Arun Suresh <ar...@gmail.com>
> wrote:
>
>> It looks like jobs that land on H9 are all getting stuck at:
>>
>> ...
>>
>> Untagged images:
>> Attempting to remove docker image sha256:4d4f8bf49bcfb5cea087e6bd6008e804f5f28a994b2dc513c2e5c4017679f8f7
>>
>>
>> Can we exclude H9 as well (or maybe fix it somehow) ?
>>
>> Cheers
>> -Arun
>>
>>
>> On Wed, Dec 21, 2016 at 12:22 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Exclusion has already been done.
>>>
>>> Precommit build #14423 has completed.
>>>
>>> On Wed, Dec 21, 2016 at 12:02 PM, Arun Suresh <ar...@gmail.com>
>>> wrote:
>>>
>>>> Actually I found H6 working yesterday and H5 bad,
>>>>
>>>> >  consider excluding H6 from Precommit builds until this problem
>>>> Can someone point me to how this can be done (I am assuming this is
>>>> some sort of exclusion list) before kicking of Jenkins job ?
>>>>
>>>> Cheers
>>>> -Arun
>>>>
>>>>
>>>>
>>>> On Wed, Dec 21, 2016 at 9:09 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> All these builds took place on :
>>>>>
>>>>> https://builds.apache.org/computer/H6
>>>>>
>>>>> Maybe log an INFRA ticket so that some admin can check what happens to
>>>>> hadoop-common-project/hadoop-common/target/test/data/4
>>>>>
>>>>> Meanwhile, consider excluding H6 from Precommit builds until this
>>>>> problem
>>>>> is solved.
>>>>>
>>>>> On Wed, Dec 21, 2016 at 9:02 AM, Sangjin Lee <sj...@apache.org> wrote:
>>>>>
>>>>> > It appears that the YARN precommit builds are broken for the most
>>>>> part. I
>>>>> > see quick failures with the following message:
>>>>> >
>>>>> > FATAL: Command "git clean -fdx" returned status code 1:
>>>>> > stdout:
>>>>> > stderr: warning: failed to remove
>>>>> > hadoop-common-project/hadoop-common/target/test/data/4
>>>>> > hudson.plugins.git.GitException
>>>>> > <http://stacktrace.jenkins-ci.org/search?query=hudson.
>>>>> > plugins.git.GitException>:
>>>>> > Command "git clean -fdx" returned status code 1:
>>>>> >
>>>>> >
>>>>> > Any ideas what might be going on?
>>>>> >
>>>>> > https://builds.apache.org/job/PreCommit-YARN-Build/
>>>>> >
>>>>> > Thanks,
>>>>> > Sangjin
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>

Re: YARN precommit busted?

[Sean Busbey <bu...@cloudera.com>]

File HADOOP-13951. will follow up there.

On Tue, Jan 3, 2017 at 3:11 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi Sean,
>
> I was trying to investigate the issue yesterday as well. I only found
> existing JIRA to track the issue is on windows:
> https://groups.google.com/forum/#!topic/jenkinsci-issues/BgcrxQqObj4.
>
> I cannot find HADOOP/YARN JIRA, please feel free to create one.
>
> Thanks,
> Wangda
>
>
> On Tue, Jan 3, 2017 at 1:04 PM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>> historically that git clean failure thing was a problem where an test
>> purposefully set bad permissions to simulate a disk failure that
>> resulted in a ro filesystem. I had thought all of the hadoop-related
>> precommit jobs had special handling in place to account for this test
>> now.
>>
>> Is there a JIRA in HADOOP or YARN tracking this issue yet? I can chase
>> down the problem pretty fast if it's just that old problem.
>>
>> On Tue, Jan 3, 2017 at 2:38 PM, Wangda Tan <wh...@gmail.com> wrote:
>> > Thanks Ted!
>> >
>> > On Tue, Jan 3, 2017 at 12:14 PM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> >> With continuous failed QA runs on H5 and H6, I have to take them out
>> >> again.
>> >>
>> >> The INFRA has bounced these two machines. Unfortunately that didn't
>> >> solve
>> >> the issue.
>> >>
>> >> On Fri, Dec 30, 2016 at 2:42 PM, Ted Yu <yu...@gmail.com> wrote:
>> >>
>> >> > Put back H5 and H6 since the INFRA ticket has been resolved.
>> >> > Excluded H9.
>> >> >
>> >> > On Fri, Dec 30, 2016 at 2:36 PM, Arun Suresh <ar...@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> It looks like jobs that land on H9 are all getting stuck at:
>> >> >>
>> >> >> ...
>> >> >>
>> >> >> Untagged images:
>> >> >> Attempting to remove docker image
>> >> >> sha256:4d4f8bf49bcfb5cea087e6bd6008e8
>> >> 04f5f28a994b2dc513c2e5c4017679f8f7
>> >> >>
>> >> >>
>> >> >> Can we exclude H9 as well (or maybe fix it somehow) ?
>> >> >>
>> >> >> Cheers
>> >> >> -Arun
>> >> >>
>> >> >>
>> >> >> On Wed, Dec 21, 2016 at 12:22 PM, Ted Yu <yu...@gmail.com>
>> >> >> wrote:
>> >> >>
>> >> >>> Exclusion has already been done.
>> >> >>>
>> >> >>> Precommit build #14423 has completed.
>> >> >>>
>> >> >>> On Wed, Dec 21, 2016 at 12:02 PM, Arun Suresh
>> >> >>> <ar...@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Actually I found H6 working yesterday and H5 bad,
>> >> >>>>
>> >> >>>> >  consider excluding H6 from Precommit builds until this problem
>> >> >>>> Can someone point me to how this can be done (I am assuming this
>> >> >>>> is
>> >> >>>> some sort of exclusion list) before kicking of Jenkins job ?
>> >> >>>>
>> >> >>>> Cheers
>> >> >>>> -Arun
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> On Wed, Dec 21, 2016 at 9:09 AM, Ted Yu <yu...@gmail.com>
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>>> All these builds took place on :
>> >> >>>>>
>> >> >>>>> https://builds.apache.org/computer/H6
>> >> >>>>>
>> >> >>>>> Maybe log an INFRA ticket so that some admin can check what
>> >> >>>>> happens
>> >> to
>> >> >>>>> hadoop-common-project/hadoop-common/target/test/data/4
>> >> >>>>>
>> >> >>>>> Meanwhile, consider excluding H6 from Precommit builds until this
>> >> >>>>> problem
>> >> >>>>> is solved.
>> >> >>>>>
>> >> >>>>> On Wed, Dec 21, 2016 at 9:02 AM, Sangjin Lee <sj...@apache.org>
>> >> wrote:
>> >> >>>>>
>> >> >>>>> > It appears that the YARN precommit builds are broken for the
>> >> >>>>> > most
>> >> >>>>> part. I
>> >> >>>>> > see quick failures with the following message:
>> >> >>>>> >
>> >> >>>>> > FATAL: Command "git clean -fdx" returned status code 1:
>> >> >>>>> > stdout:
>> >> >>>>> > stderr: warning: failed to remove
>> >> >>>>> > hadoop-common-project/hadoop-common/target/test/data/4
>> >> >>>>> > hudson.plugins.git.GitException
>> >> >>>>> > <http://stacktrace.jenkins-ci.org/search?query=hudson.
>> >> >>>>> > plugins.git.GitException>:
>> >> >>>>> > Command "git clean -fdx" returned status code 1:
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > Any ideas what might be going on?
>> >> >>>>> >
>> >> >>>>> > https://builds.apache.org/job/PreCommit-YARN-Build/
>> >> >>>>> >
>> >> >>>>> > Thanks,
>> >> >>>>> > Sangjin
>> >> >>>>> >
>> >> >>>>>
>> >> >>>>
>> >> >>>>
>> >> >>>
>> >> >>
>> >> >
>> >>
>>
>>
>>
>> --
>> busbey
>
>



-- 
busbey

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: YARN precommit busted?

[Wangda Tan <wh...@gmail.com>]

Hi Sean,

I was trying to investigate the issue yesterday as well. I only found
existing JIRA to track the issue is on windows:
https://groups.google.com/forum/#!topic/jenkinsci-issues/BgcrxQqObj4.

I cannot find HADOOP/YARN JIRA, please feel free to create one.

Thanks,
Wangda


On Tue, Jan 3, 2017 at 1:04 PM, Sean Busbey <bu...@cloudera.com> wrote:

> historically that git clean failure thing was a problem where an test
> purposefully set bad permissions to simulate a disk failure that
> resulted in a ro filesystem. I had thought all of the hadoop-related
> precommit jobs had special handling in place to account for this test
> now.
>
> Is there a JIRA in HADOOP or YARN tracking this issue yet? I can chase
> down the problem pretty fast if it's just that old problem.
>
> On Tue, Jan 3, 2017 at 2:38 PM, Wangda Tan <wh...@gmail.com> wrote:
> > Thanks Ted!
> >
> > On Tue, Jan 3, 2017 at 12:14 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> With continuous failed QA runs on H5 and H6, I have to take them out
> again.
> >>
> >> The INFRA has bounced these two machines. Unfortunately that didn't
> solve
> >> the issue.
> >>
> >> On Fri, Dec 30, 2016 at 2:42 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > Put back H5 and H6 since the INFRA ticket has been resolved.
> >> > Excluded H9.
> >> >
> >> > On Fri, Dec 30, 2016 at 2:36 PM, Arun Suresh <ar...@gmail.com>
> >> > wrote:
> >> >
> >> >> It looks like jobs that land on H9 are all getting stuck at:
> >> >>
> >> >> ...
> >> >>
> >> >> Untagged images:
> >> >> Attempting to remove docker image sha256:
> 4d4f8bf49bcfb5cea087e6bd6008e8
> >> 04f5f28a994b2dc513c2e5c4017679f8f7
> >> >>
> >> >>
> >> >> Can we exclude H9 as well (or maybe fix it somehow) ?
> >> >>
> >> >> Cheers
> >> >> -Arun
> >> >>
> >> >>
> >> >> On Wed, Dec 21, 2016 at 12:22 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >> >>
> >> >>> Exclusion has already been done.
> >> >>>
> >> >>> Precommit build #14423 has completed.
> >> >>>
> >> >>> On Wed, Dec 21, 2016 at 12:02 PM, Arun Suresh <
> arun.suresh@gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>>> Actually I found H6 working yesterday and H5 bad,
> >> >>>>
> >> >>>> >  consider excluding H6 from Precommit builds until this problem
> >> >>>> Can someone point me to how this can be done (I am assuming this is
> >> >>>> some sort of exclusion list) before kicking of Jenkins job ?
> >> >>>>
> >> >>>> Cheers
> >> >>>> -Arun
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Wed, Dec 21, 2016 at 9:09 AM, Ted Yu <yu...@gmail.com>
> wrote:
> >> >>>>
> >> >>>>> All these builds took place on :
> >> >>>>>
> >> >>>>> https://builds.apache.org/computer/H6
> >> >>>>>
> >> >>>>> Maybe log an INFRA ticket so that some admin can check what
> happens
> >> to
> >> >>>>> hadoop-common-project/hadoop-common/target/test/data/4
> >> >>>>>
> >> >>>>> Meanwhile, consider excluding H6 from Precommit builds until this
> >> >>>>> problem
> >> >>>>> is solved.
> >> >>>>>
> >> >>>>> On Wed, Dec 21, 2016 at 9:02 AM, Sangjin Lee <sj...@apache.org>
> >> wrote:
> >> >>>>>
> >> >>>>> > It appears that the YARN precommit builds are broken for the
> most
> >> >>>>> part. I
> >> >>>>> > see quick failures with the following message:
> >> >>>>> >
> >> >>>>> > FATAL: Command "git clean -fdx" returned status code 1:
> >> >>>>> > stdout:
> >> >>>>> > stderr: warning: failed to remove
> >> >>>>> > hadoop-common-project/hadoop-common/target/test/data/4
> >> >>>>> > hudson.plugins.git.GitException
> >> >>>>> > <http://stacktrace.jenkins-ci.org/search?query=hudson.
> >> >>>>> > plugins.git.GitException>:
> >> >>>>> > Command "git clean -fdx" returned status code 1:
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > Any ideas what might be going on?
> >> >>>>> >
> >> >>>>> > https://builds.apache.org/job/PreCommit-YARN-Build/
> >> >>>>> >
> >> >>>>> > Thanks,
> >> >>>>> > Sangjin
> >> >>>>> >
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>
> >> >>
> >> >
> >>
>
>
>
> --
> busbey
>

Re: YARN precommit busted?

[Sean Busbey <bu...@cloudera.com>]

historically that git clean failure thing was a problem where an test
purposefully set bad permissions to simulate a disk failure that
resulted in a ro filesystem. I had thought all of the hadoop-related
precommit jobs had special handling in place to account for this test
now.

Is there a JIRA in HADOOP or YARN tracking this issue yet? I can chase
down the problem pretty fast if it's just that old problem.

On Tue, Jan 3, 2017 at 2:38 PM, Wangda Tan <wh...@gmail.com> wrote:
> Thanks Ted!
>
> On Tue, Jan 3, 2017 at 12:14 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> With continuous failed QA runs on H5 and H6, I have to take them out again.
>>
>> The INFRA has bounced these two machines. Unfortunately that didn't solve
>> the issue.
>>
>> On Fri, Dec 30, 2016 at 2:42 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > Put back H5 and H6 since the INFRA ticket has been resolved.
>> > Excluded H9.
>> >
>> > On Fri, Dec 30, 2016 at 2:36 PM, Arun Suresh <ar...@gmail.com>
>> > wrote:
>> >
>> >> It looks like jobs that land on H9 are all getting stuck at:
>> >>
>> >> ...
>> >>
>> >> Untagged images:
>> >> Attempting to remove docker image sha256:4d4f8bf49bcfb5cea087e6bd6008e8
>> 04f5f28a994b2dc513c2e5c4017679f8f7
>> >>
>> >>
>> >> Can we exclude H9 as well (or maybe fix it somehow) ?
>> >>
>> >> Cheers
>> >> -Arun
>> >>
>> >>
>> >> On Wed, Dec 21, 2016 at 12:22 PM, Ted Yu <yu...@gmail.com> wrote:
>> >>
>> >>> Exclusion has already been done.
>> >>>
>> >>> Precommit build #14423 has completed.
>> >>>
>> >>> On Wed, Dec 21, 2016 at 12:02 PM, Arun Suresh <ar...@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Actually I found H6 working yesterday and H5 bad,
>> >>>>
>> >>>> >  consider excluding H6 from Precommit builds until this problem
>> >>>> Can someone point me to how this can be done (I am assuming this is
>> >>>> some sort of exclusion list) before kicking of Jenkins job ?
>> >>>>
>> >>>> Cheers
>> >>>> -Arun
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Wed, Dec 21, 2016 at 9:09 AM, Ted Yu <yu...@gmail.com> wrote:
>> >>>>
>> >>>>> All these builds took place on :
>> >>>>>
>> >>>>> https://builds.apache.org/computer/H6
>> >>>>>
>> >>>>> Maybe log an INFRA ticket so that some admin can check what happens
>> to
>> >>>>> hadoop-common-project/hadoop-common/target/test/data/4
>> >>>>>
>> >>>>> Meanwhile, consider excluding H6 from Precommit builds until this
>> >>>>> problem
>> >>>>> is solved.
>> >>>>>
>> >>>>> On Wed, Dec 21, 2016 at 9:02 AM, Sangjin Lee <sj...@apache.org>
>> wrote:
>> >>>>>
>> >>>>> > It appears that the YARN precommit builds are broken for the most
>> >>>>> part. I
>> >>>>> > see quick failures with the following message:
>> >>>>> >
>> >>>>> > FATAL: Command "git clean -fdx" returned status code 1:
>> >>>>> > stdout:
>> >>>>> > stderr: warning: failed to remove
>> >>>>> > hadoop-common-project/hadoop-common/target/test/data/4
>> >>>>> > hudson.plugins.git.GitException
>> >>>>> > <http://stacktrace.jenkins-ci.org/search?query=hudson.
>> >>>>> > plugins.git.GitException>:
>> >>>>> > Command "git clean -fdx" returned status code 1:
>> >>>>> >
>> >>>>> >
>> >>>>> > Any ideas what might be going on?
>> >>>>> >
>> >>>>> > https://builds.apache.org/job/PreCommit-YARN-Build/
>> >>>>> >
>> >>>>> > Thanks,
>> >>>>> > Sangjin
>> >>>>> >
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>



-- 
busbey

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: YARN precommit busted?

[Wangda Tan <wh...@gmail.com>]

Thanks Ted!

On Tue, Jan 3, 2017 at 12:14 PM, Ted Yu <yu...@gmail.com> wrote:

> With continuous failed QA runs on H5 and H6, I have to take them out again.
>
> The INFRA has bounced these two machines. Unfortunately that didn't solve
> the issue.
>
> On Fri, Dec 30, 2016 at 2:42 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Put back H5 and H6 since the INFRA ticket has been resolved.
> > Excluded H9.
> >
> > On Fri, Dec 30, 2016 at 2:36 PM, Arun Suresh <ar...@gmail.com>
> > wrote:
> >
> >> It looks like jobs that land on H9 are all getting stuck at:
> >>
> >> ...
> >>
> >> Untagged images:
> >> Attempting to remove docker image sha256:4d4f8bf49bcfb5cea087e6bd6008e8
> 04f5f28a994b2dc513c2e5c4017679f8f7
> >>
> >>
> >> Can we exclude H9 as well (or maybe fix it somehow) ?
> >>
> >> Cheers
> >> -Arun
> >>
> >>
> >> On Wed, Dec 21, 2016 at 12:22 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> Exclusion has already been done.
> >>>
> >>> Precommit build #14423 has completed.
> >>>
> >>> On Wed, Dec 21, 2016 at 12:02 PM, Arun Suresh <ar...@gmail.com>
> >>> wrote:
> >>>
> >>>> Actually I found H6 working yesterday and H5 bad,
> >>>>
> >>>> >  consider excluding H6 from Precommit builds until this problem
> >>>> Can someone point me to how this can be done (I am assuming this is
> >>>> some sort of exclusion list) before kicking of Jenkins job ?
> >>>>
> >>>> Cheers
> >>>> -Arun
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Dec 21, 2016 at 9:09 AM, Ted Yu <yu...@gmail.com> wrote:
> >>>>
> >>>>> All these builds took place on :
> >>>>>
> >>>>> https://builds.apache.org/computer/H6
> >>>>>
> >>>>> Maybe log an INFRA ticket so that some admin can check what happens
> to
> >>>>> hadoop-common-project/hadoop-common/target/test/data/4
> >>>>>
> >>>>> Meanwhile, consider excluding H6 from Precommit builds until this
> >>>>> problem
> >>>>> is solved.
> >>>>>
> >>>>> On Wed, Dec 21, 2016 at 9:02 AM, Sangjin Lee <sj...@apache.org>
> wrote:
> >>>>>
> >>>>> > It appears that the YARN precommit builds are broken for the most
> >>>>> part. I
> >>>>> > see quick failures with the following message:
> >>>>> >
> >>>>> > FATAL: Command "git clean -fdx" returned status code 1:
> >>>>> > stdout:
> >>>>> > stderr: warning: failed to remove
> >>>>> > hadoop-common-project/hadoop-common/target/test/data/4
> >>>>> > hudson.plugins.git.GitException
> >>>>> > <http://stacktrace.jenkins-ci.org/search?query=hudson.
> >>>>> > plugins.git.GitException>:
> >>>>> > Command "git clean -fdx" returned status code 1:
> >>>>> >
> >>>>> >
> >>>>> > Any ideas what might be going on?
> >>>>> >
> >>>>> > https://builds.apache.org/job/PreCommit-YARN-Build/
> >>>>> >
> >>>>> > Thanks,
> >>>>> > Sangjin
> >>>>> >
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >
>