You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Nigel Daley <nd...@mac.com> on 2010/10/20 21:47:15 UTC
Patch testing
Folks,
I'm working to get the pre-commit patch testing running again for HDFS, HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from day-to-day involvement w/ the project over the last 6 months (new job), I want to check in and make sure folks would still find this pre-commit testing useful. Also, happy to hear any suggested improvement. Let me know.
Cheers,
Nige
Re: Patch testing
Posted by Todd Lipcon <to...@cloudera.com>.
On Fri, Dec 17, 2010 at 3:55 PM, Jakob Homan <jg...@gmail.com> wrote:
>> If HDFS is added to the test-patch queue right now we get
>> nothing but dozens of -1'ed patches.
> There aren't dozens of patches being submitted currently. The -1
> isn't the important thing, it's the grunt work of actually running
> (and waiting) for the tests, test-patch, etc. that Hudson does so that
> the developer doesn't have to.
>
I agree with Jakob. I've had to run and re-run the test-patch and unit
tests probably 30 times over the last two weeks, and it takes a lot of
effort, since my own infrastructure for doing this is a bit messy. I'd
much rather just reply to the Hudson comments saying "these are known
issues" than have to run the tests, check the results, copy and paste
them and *then* say "these are known issues" anyway!
> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
>> +1, thanks for doing this.
>>
>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
>>
>>> So, with test-patch updated to show the failing tests, saving the
>>> developers the need to go and verify that the failed tests are all
>>> known, how do people feel about turning on test-patch again for HDFS
>>> and mapred? I think it'll help prevent any more tests from entering
>>> the "yeah, we know" category.
>>>
>>> Thanks,
>>> jg
>>>
>>>
>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>>> > True, each patch would get a -1 and the failing tests would need to be
>>> > verified as those known bad (BTW, it would be great if Hudson could list
>>> > which tests failed in the message it posts to JIRA). But that's still
>>> quite
>>> > a bit less error-prone work than if the developer runs the tests and
>>> > test-patch themselves. Also, with 22 being cut, there are a lot of
>>> patches
>>> > up in the air and several developers are juggling multiple patches. The
>>> > more automation we can have, even if it's not perfect, will decrease
>>> errors
>>> > we may make.
>>> > -jg
>>> >
>>> > Nigel Daley wrote:
>>> >>
>>> >> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>> >>
>>> >>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>> >>>> until these projects build and test cleanly. Looks like both these
>>> projects
>>> >>>> currently have test failures.
>>> >>>
>>> >>> Assuming the projects are compiling and building, is there a reason to
>>> >>> not turn it on despite the test failures? Hudson is invaluable to
>>> developers
>>> >>> who then don't have to run the tests and test-patch themselves. We
>>> didn't
>>> >>> turn Hudson off when it was working previously and there were known
>>> >>> failures. I think one of the reasons we have more failing tests now is
>>> the
>>> >>> higher cost of doing Hudson's work (not a great excuse I know). This
>>> is
>>> >>> particularly true now because several of the failing tests involve
>>> tests
>>> >>> timing out, making the whole testing regime even longer.
>>> >>
>>> >> Every single patch would get a -1 and need investigation. Currently,
>>> that
>>> >> would be about 83 investigations between MR and HDFS issues that are in
>>> >> patch available state. Shouldn't we focus on getting these tests fixed
>>> or
>>> >> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>> >> well) before I turn this on.
>>> >>
>>> >> Cheers,
>>> >> Nige
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Patch testing
Posted by Todd Lipcon <to...@cloudera.com>.
On Wed, Jan 26, 2011 at 10:05 AM, Nigel Daley <nd...@mac.com> wrote:
> raid (contrib) test hanging: TestBlockFixer
>
> I forced 2 thread dumps. Both hung in the same place. Filed
> https://issues.apache.org/jira/browse/MAPREDUCE-2283 This is a blocker
> for turning on MR precommit.
>
Since this is contrib, I'd like to suggest just disabling this test
temporarily. We can re-enable it once it's fixed.
Not having MR pre-commit working has been pretty painful.
-Todd
> On Jan 25, 2011, at 11:19 PM, Nigel Daley wrote:
>
> > Started another trial run of MR precommit testing:
> >
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/17/
> >
> > Let's see if 17th time is a charm...
> >
> > Nige
> >
> > On Jan 7, 2011, at 5:14 PM, Todd Lipcon wrote:
> >
> >> On Fri, Jan 7, 2011 at 2:11 PM, Nigel Daley <nd...@mac.com> wrote:
> >>
> >>> Hrm, the MR precommit test I'm running has hung (been running for 14
> hours
> >>> so far). FWIW, 2 HDFS precommit tests are hung too. I suspect it
> could be
> >>> the NFS mounts on the machines. I forced a thread dump which you can
> see in
> >>> the console:
> >>>
> https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/10/console
> >>>
> >>>
> >> Strange, haven't seen a hang like that before in
> handleConnectionFailure. It
> >> should retry for 15 minutes max in that loop.
> >>
> >>
> >>> Any other ideas why these might be hanging?
> >>>
> >>>
> >> There is an HDFS bug right now that can cause hangs on some tests -
> >> HDFS-1529 - would appreciate if someone can take a look. But I don't
> think
> >> this is responsible for the MR hang above.
> >>
> >> -Todd
> >>
> >>
> >>> On Jan 5, 2011, at 5:42 PM, Todd Lipcon wrote:
> >>>
> >>>> On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <nd...@mac.com> wrote:
> >>>>
> >>>>> Thanks for looking into it Todd. Let's first see if you think it can
> be
> >>>>> fixed quickly. Let me know.
> >>>>>
> >>>>>
> >>>> No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which
> >>> fixes
> >>>> this test timeout for me.
> >>>>
> >>>> -Todd
> >>>>
> >>>>
> >>>>> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
> >>>>>
> >>>>>> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <nd...@mac.com> wrote:
> >>>>>>
> >>>>>>> Todd, would love to get
> >>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first
> >>> since
> >>>>>>> this is failing every night on trunk.
> >>>>>>>
> >>>>>>
> >>>>>> What if we disable that test, move that issue to 0.22 blocker, and
> then
> >>>>>> enable the test-patch? I'll also look into that one today, but if
> it's
> >>>>>> something that will take a while to fix, I don't think we should
> hold
> >>> off
> >>>>>> the useful testing for all the other patches.
> >>>>>>
> >>>>>> -Todd
> >>>>>>
> >>>>>> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
> >>>>>>>
> >>>>>>>> Hi Nigel,
> >>>>>>>>
> >>>>>>>> MAPREDUCE-2172 has been fixed for a while. Are there any other
> >>>>> particular
> >>>>>>>> JIRAs you think need to be fixed before the MR test-patch queue
> gets
> >>>>>>>> enabled? I have a lot of outstanding patches and doing all the
> >>>>> test-patch
> >>>>>>>> turnaround manually on 3 different boxes is a real headache.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> -Todd
> >>>>>>>>
> >>>>>>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <nd...@mac.com>
> wrote:
> >>>>>>>>
> >>>>>>>>> Ok, HDFS is now enabled. You'll see a stream of updates shortly
> on
> >>>>> the
> >>>>>>> ~30
> >>>>>>>>> Patch Available HDFS issues.
> >>>>>>>>>
> >>>>>>>>> Nige
> >>>>>>>>>
> >>>>>>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
> >>>>>>>>>
> >>>>>>>>>> I committed HDFS-1511 this morning. We should be good to go. I
> >>> can
> >>>>>>>>>> haz snooty robot butler?
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <
> >>> cos@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I
> think,
> >>>>>>>>>>> unless it is done earlier.
> >>>>>>>>>>> --
> >>>>>>>>>>> Take care,
> >>>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>> Ok. I'll get a patch out for 1511 tomorrow, unless someone
> wants
> >>>>> to
> >>>>>>>>>>>> whip one up tonight.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com>
> >>>>> wrote:
> >>>>>>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done
> >>> I'll
> >>>>>>>>> enable hdfs patch testing.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>> Nige
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Sent from my iPhone4
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <
> cos@apache.org
> >>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> One more issue needs to be addressed before test-patch is
> >>> turned
> >>>>> on
> >>>>>>>>> HDFS is
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Take care,
> >>>>>>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
> >>>>> cos@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>> Considering that because of these 4 faulty cases every
> patch
> >>>>> will
> >>>>>>> be
> >>>>>>>>>>>>>>> -1'ed a patch author will still have to look at it and make
> a
> >>>>>>>>> comment
> >>>>>>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps,
> but
> >>>>>>>>> messier
> >>>>>>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a
> better
> >>>>> way.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> Take care,
> >>>>>>>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <
> jghoman@gmail.com
> >>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
> >>>>>>>>>>>>>>>>> nothing but dozens of -1'ed patches.
> >>>>>>>>>>>>>>>> There aren't dozens of patches being submitted currently.
> >>> The
> >>>>> -1
> >>>>>>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
> >>>>>>> running
> >>>>>>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson
> >>> does
> >>>>> so
> >>>>>>>>> that
> >>>>>>>>>>>>>>>> the developer doesn't have to.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
> >>>>>>>>> dhruba@gmail.com> wrote:
> >>>>>>>>>>>>>>>>> +1, thanks for doing this.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
> >>>>> jghoman@gmail.com
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> So, with test-patch updated to show the failing tests,
> >>> saving
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> developers the need to go and verify that the failed
> tests
> >>>>> are
> >>>>>>>>> all
> >>>>>>>>>>>>>>>>>> known, how do people feel about turning on test-patch
> again
> >>>>> for
> >>>>>>>>> HDFS
> >>>>>>>>>>>>>>>>>> and mapred? I think it'll help prevent any more tests
> from
> >>>>>>>>> entering
> >>>>>>>>>>>>>>>>>> the "yeah, we know" category.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> jg
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
> >>>>>>>>> jhoman@yahoo-inc.com> wrote:
> >>>>>>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests
> >>> would
> >>>>>>> need
> >>>>>>>>> to be
> >>>>>>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
> >>>>> Hudson
> >>>>>>>>> could list
> >>>>>>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA).
> But
> >>>>>>> that's
> >>>>>>>>> still
> >>>>>>>>>>>>>>>>>> quite
> >>>>>>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs
> the
> >>>>>>> tests
> >>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> test-patch themselves. Also, with 22 being cut, there
> are
> >>> a
> >>>>>>> lot
> >>>>>>>>> of
> >>>>>>>>>>>>>>>>>> patches
> >>>>>>>>>>>>>>>>>>> up in the air and several developers are juggling
> multiple
> >>>>>>>>> patches. The
> >>>>>>>>>>>>>>>>>>> more automation we can have, even if it's not perfect,
> >>> will
> >>>>>>>>> decrease
> >>>>>>>>>>>>>>>>>> errors
> >>>>>>>>>>>>>>>>>>> we may make.
> >>>>>>>>>>>>>>>>>>> -jg
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Nigel Daley wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we
> >>> won't
> >>>>>>>>> turn it on
> >>>>>>>>>>>>>>>>>>>>>> until these projects build and test cleanly. Looks
> >>> like
> >>>>>>> both
> >>>>>>>>> these
> >>>>>>>>>>>>>>>>>> projects
> >>>>>>>>>>>>>>>>>>>>>> currently have test failures.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is
> >>> there
> >>>>> a
> >>>>>>>>> reason to
> >>>>>>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
> >>>>>>> invaluable
> >>>>>>>>> to
> >>>>>>>>>>>>>>>>>> developers
> >>>>>>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
> >>>>>>>>> themselves. We
> >>>>>>>>>>>>>>>>>> didn't
> >>>>>>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and
> there
> >>>>>>> were
> >>>>>>>>> known
> >>>>>>>>>>>>>>>>>>>>> failures. I think one of the reasons we have more
> >>> failing
> >>>>>>>>> tests now is
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great
> excuse I
> >>>>>>>>> know). This
> >>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> particularly true now because several of the failing
> >>> tests
> >>>>>>>>> involve
> >>>>>>>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>>>>>>> timing out, making the whole testing regime even
> longer.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Every single patch would get a -1 and need
> investigation.
> >>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS
> >>> issues
> >>>>>>>>> that are in
> >>>>>>>>>>>>>>>>>>>> patch available state. Shouldn't we focus on getting
> >>> these
> >>>>>>>>> tests fixed
> >>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed
> >>>>> (applies
> >>>>>>> to
> >>>>>>>>> HDFS as
> >>>>>>>>>>>>>>>>>>>> well) before I turn this on.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>> Nige
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Todd Lipcon
> >>>>>>>> Software Engineer, Cloudera
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Todd Lipcon
> >>>>>> Software Engineer, Cloudera
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Todd Lipcon
> >>>> Software Engineer, Cloudera
> >>>
> >>>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
raid (contrib) test hanging: TestBlockFixer
I forced 2 thread dumps. Both hung in the same place. Filed https://issues.apache.org/jira/browse/MAPREDUCE-2283 This is a blocker for turning on MR precommit.
Cheers,
Nige
On Jan 25, 2011, at 11:19 PM, Nigel Daley wrote:
> Started another trial run of MR precommit testing:
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/17/
>
> Let's see if 17th time is a charm...
>
> Nige
>
> On Jan 7, 2011, at 5:14 PM, Todd Lipcon wrote:
>
>> On Fri, Jan 7, 2011 at 2:11 PM, Nigel Daley <nd...@mac.com> wrote:
>>
>>> Hrm, the MR precommit test I'm running has hung (been running for 14 hours
>>> so far). FWIW, 2 HDFS precommit tests are hung too. I suspect it could be
>>> the NFS mounts on the machines. I forced a thread dump which you can see in
>>> the console:
>>> https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/10/console
>>>
>>>
>> Strange, haven't seen a hang like that before in handleConnectionFailure. It
>> should retry for 15 minutes max in that loop.
>>
>>
>>> Any other ideas why these might be hanging?
>>>
>>>
>> There is an HDFS bug right now that can cause hangs on some tests -
>> HDFS-1529 - would appreciate if someone can take a look. But I don't think
>> this is responsible for the MR hang above.
>>
>> -Todd
>>
>>
>>> On Jan 5, 2011, at 5:42 PM, Todd Lipcon wrote:
>>>
>>>> On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <nd...@mac.com> wrote:
>>>>
>>>>> Thanks for looking into it Todd. Let's first see if you think it can be
>>>>> fixed quickly. Let me know.
>>>>>
>>>>>
>>>> No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which
>>> fixes
>>>> this test timeout for me.
>>>>
>>>> -Todd
>>>>
>>>>
>>>>> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
>>>>>
>>>>>> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <nd...@mac.com> wrote:
>>>>>>
>>>>>>> Todd, would love to get
>>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first
>>> since
>>>>>>> this is failing every night on trunk.
>>>>>>>
>>>>>>
>>>>>> What if we disable that test, move that issue to 0.22 blocker, and then
>>>>>> enable the test-patch? I'll also look into that one today, but if it's
>>>>>> something that will take a while to fix, I don't think we should hold
>>> off
>>>>>> the useful testing for all the other patches.
>>>>>>
>>>>>> -Todd
>>>>>>
>>>>>> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
>>>>>>>
>>>>>>>> Hi Nigel,
>>>>>>>>
>>>>>>>> MAPREDUCE-2172 has been fixed for a while. Are there any other
>>>>> particular
>>>>>>>> JIRAs you think need to be fixed before the MR test-patch queue gets
>>>>>>>> enabled? I have a lot of outstanding patches and doing all the
>>>>> test-patch
>>>>>>>> turnaround manually on 3 different boxes is a real headache.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <nd...@mac.com> wrote:
>>>>>>>>
>>>>>>>>> Ok, HDFS is now enabled. You'll see a stream of updates shortly on
>>>>> the
>>>>>>> ~30
>>>>>>>>> Patch Available HDFS issues.
>>>>>>>>>
>>>>>>>>> Nige
>>>>>>>>>
>>>>>>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>>>>>>>>>
>>>>>>>>>> I committed HDFS-1511 this morning. We should be good to go. I
>>> can
>>>>>>>>>> haz snooty robot butler?
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <
>>> cos@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>>>>>>>>>>> unless it is done earlier.
>>>>>>>>>>> --
>>>>>>>>>>> Take care,
>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com>
>>>>> wrote:
>>>>>>>>>>>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants
>>>>> to
>>>>>>>>>>>> whip one up tonight.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com>
>>>>> wrote:
>>>>>>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done
>>> I'll
>>>>>>>>> enable hdfs patch testing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Nige
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sent from my iPhone4
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org
>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> One more issue needs to be addressed before test-patch is
>>> turned
>>>>> on
>>>>>>>>> HDFS is
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Take care,
>>>>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
>>>>> cos@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Considering that because of these 4 faulty cases every patch
>>>>> will
>>>>>>> be
>>>>>>>>>>>>>>> -1'ed a patch author will still have to look at it and make a
>>>>>>>>> comment
>>>>>>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
>>>>>>>>> messier
>>>>>>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better
>>>>> way.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Take care,
>>>>>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com
>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>>>>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>>>>>>>>>>> There aren't dozens of patches being submitted currently.
>>> The
>>>>> -1
>>>>>>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
>>>>>>> running
>>>>>>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson
>>> does
>>>>> so
>>>>>>>>> that
>>>>>>>>>>>>>>>> the developer doesn't have to.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
>>>>>>>>> dhruba@gmail.com> wrote:
>>>>>>>>>>>>>>>>> +1, thanks for doing this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
>>>>> jghoman@gmail.com
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So, with test-patch updated to show the failing tests,
>>> saving
>>>>>>> the
>>>>>>>>>>>>>>>>>> developers the need to go and verify that the failed tests
>>>>> are
>>>>>>>>> all
>>>>>>>>>>>>>>>>>> known, how do people feel about turning on test-patch again
>>>>> for
>>>>>>>>> HDFS
>>>>>>>>>>>>>>>>>> and mapred? I think it'll help prevent any more tests from
>>>>>>>>> entering
>>>>>>>>>>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> jg
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
>>>>>>>>> jhoman@yahoo-inc.com> wrote:
>>>>>>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests
>>> would
>>>>>>> need
>>>>>>>>> to be
>>>>>>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
>>>>> Hudson
>>>>>>>>> could list
>>>>>>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA). But
>>>>>>> that's
>>>>>>>>> still
>>>>>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
>>>>>>> tests
>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are
>>> a
>>>>>>> lot
>>>>>>>>> of
>>>>>>>>>>>>>>>>>> patches
>>>>>>>>>>>>>>>>>>> up in the air and several developers are juggling multiple
>>>>>>>>> patches. The
>>>>>>>>>>>>>>>>>>> more automation we can have, even if it's not perfect,
>>> will
>>>>>>>>> decrease
>>>>>>>>>>>>>>>>>> errors
>>>>>>>>>>>>>>>>>>> we may make.
>>>>>>>>>>>>>>>>>>> -jg
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we
>>> won't
>>>>>>>>> turn it on
>>>>>>>>>>>>>>>>>>>>>> until these projects build and test cleanly. Looks
>>> like
>>>>>>> both
>>>>>>>>> these
>>>>>>>>>>>>>>>>>> projects
>>>>>>>>>>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is
>>> there
>>>>> a
>>>>>>>>> reason to
>>>>>>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
>>>>>>> invaluable
>>>>>>>>> to
>>>>>>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
>>>>>>>>> themselves. We
>>>>>>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
>>>>>>> were
>>>>>>>>> known
>>>>>>>>>>>>>>>>>>>>> failures. I think one of the reasons we have more
>>> failing
>>>>>>>>> tests now is
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
>>>>>>>>> know). This
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> particularly true now because several of the failing
>>> tests
>>>>>>>>> involve
>>>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS
>>> issues
>>>>>>>>> that are in
>>>>>>>>>>>>>>>>>>>> patch available state. Shouldn't we focus on getting
>>> these
>>>>>>>>> tests fixed
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed
>>>>> (applies
>>>>>>> to
>>>>>>>>> HDFS as
>>>>>>>>>>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> Nige
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Todd Lipcon
>>>>>>>> Software Engineer, Cloudera
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Todd Lipcon
>>>>>> Software Engineer, Cloudera
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>
>>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
Started another trial run of MR precommit testing:
https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/17/
Let's see if 17th time is a charm...
Nige
On Jan 7, 2011, at 5:14 PM, Todd Lipcon wrote:
> On Fri, Jan 7, 2011 at 2:11 PM, Nigel Daley <nd...@mac.com> wrote:
>
>> Hrm, the MR precommit test I'm running has hung (been running for 14 hours
>> so far). FWIW, 2 HDFS precommit tests are hung too. I suspect it could be
>> the NFS mounts on the machines. I forced a thread dump which you can see in
>> the console:
>> https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/10/console
>>
>>
> Strange, haven't seen a hang like that before in handleConnectionFailure. It
> should retry for 15 minutes max in that loop.
>
>
>> Any other ideas why these might be hanging?
>>
>>
> There is an HDFS bug right now that can cause hangs on some tests -
> HDFS-1529 - would appreciate if someone can take a look. But I don't think
> this is responsible for the MR hang above.
>
> -Todd
>
>
>> On Jan 5, 2011, at 5:42 PM, Todd Lipcon wrote:
>>
>>> On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <nd...@mac.com> wrote:
>>>
>>>> Thanks for looking into it Todd. Let's first see if you think it can be
>>>> fixed quickly. Let me know.
>>>>
>>>>
>>> No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which
>> fixes
>>> this test timeout for me.
>>>
>>> -Todd
>>>
>>>
>>>> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
>>>>
>>>>> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <nd...@mac.com> wrote:
>>>>>
>>>>>> Todd, would love to get
>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first
>> since
>>>>>> this is failing every night on trunk.
>>>>>>
>>>>>
>>>>> What if we disable that test, move that issue to 0.22 blocker, and then
>>>>> enable the test-patch? I'll also look into that one today, but if it's
>>>>> something that will take a while to fix, I don't think we should hold
>> off
>>>>> the useful testing for all the other patches.
>>>>>
>>>>> -Todd
>>>>>
>>>>> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
>>>>>>
>>>>>>> Hi Nigel,
>>>>>>>
>>>>>>> MAPREDUCE-2172 has been fixed for a while. Are there any other
>>>> particular
>>>>>>> JIRAs you think need to be fixed before the MR test-patch queue gets
>>>>>>> enabled? I have a lot of outstanding patches and doing all the
>>>> test-patch
>>>>>>> turnaround manually on 3 different boxes is a real headache.
>>>>>>>
>>>>>>> Thanks
>>>>>>> -Todd
>>>>>>>
>>>>>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <nd...@mac.com> wrote:
>>>>>>>
>>>>>>>> Ok, HDFS is now enabled. You'll see a stream of updates shortly on
>>>> the
>>>>>> ~30
>>>>>>>> Patch Available HDFS issues.
>>>>>>>>
>>>>>>>> Nige
>>>>>>>>
>>>>>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>>>>>>>>
>>>>>>>>> I committed HDFS-1511 this morning. We should be good to go. I
>> can
>>>>>>>>> haz snooty robot butler?
>>>>>>>>>
>>>>>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <
>> cos@apache.org>
>>>>>>>> wrote:
>>>>>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>>>>>>>>>> unless it is done earlier.
>>>>>>>>>> --
>>>>>>>>>> Take care,
>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com>
>>>> wrote:
>>>>>>>>>>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants
>>>> to
>>>>>>>>>>> whip one up tonight.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com>
>>>> wrote:
>>>>>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done
>> I'll
>>>>>>>> enable hdfs patch testing.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Nige
>>>>>>>>>>>>
>>>>>>>>>>>> Sent from my iPhone4
>>>>>>>>>>>>
>>>>>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org
>>>
>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> One more issue needs to be addressed before test-patch is
>> turned
>>>> on
>>>>>>>> HDFS is
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Take care,
>>>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
>>>> cos@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>>>> Considering that because of these 4 faulty cases every patch
>>>> will
>>>>>> be
>>>>>>>>>>>>>> -1'ed a patch author will still have to look at it and make a
>>>>>>>> comment
>>>>>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
>>>>>>>> messier
>>>>>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better
>>>> way.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Take care,
>>>>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com
>>>
>>>>>>>> wrote:
>>>>>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>>>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>>>>>>>>>> There aren't dozens of patches being submitted currently.
>> The
>>>> -1
>>>>>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
>>>>>> running
>>>>>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson
>> does
>>>> so
>>>>>>>> that
>>>>>>>>>>>>>>> the developer doesn't have to.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
>>>>>>>> dhruba@gmail.com> wrote:
>>>>>>>>>>>>>>>> +1, thanks for doing this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
>>>> jghoman@gmail.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So, with test-patch updated to show the failing tests,
>> saving
>>>>>> the
>>>>>>>>>>>>>>>>> developers the need to go and verify that the failed tests
>>>> are
>>>>>>>> all
>>>>>>>>>>>>>>>>> known, how do people feel about turning on test-patch again
>>>> for
>>>>>>>> HDFS
>>>>>>>>>>>>>>>>> and mapred? I think it'll help prevent any more tests from
>>>>>>>> entering
>>>>>>>>>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> jg
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
>>>>>>>> jhoman@yahoo-inc.com> wrote:
>>>>>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests
>> would
>>>>>> need
>>>>>>>> to be
>>>>>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
>>>> Hudson
>>>>>>>> could list
>>>>>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA). But
>>>>>> that's
>>>>>>>> still
>>>>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
>>>>>> tests
>>>>>>>> and
>>>>>>>>>>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are
>> a
>>>>>> lot
>>>>>>>> of
>>>>>>>>>>>>>>>>> patches
>>>>>>>>>>>>>>>>>> up in the air and several developers are juggling multiple
>>>>>>>> patches. The
>>>>>>>>>>>>>>>>>> more automation we can have, even if it's not perfect,
>> will
>>>>>>>> decrease
>>>>>>>>>>>>>>>>> errors
>>>>>>>>>>>>>>>>>> we may make.
>>>>>>>>>>>>>>>>>> -jg
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we
>> won't
>>>>>>>> turn it on
>>>>>>>>>>>>>>>>>>>>> until these projects build and test cleanly. Looks
>> like
>>>>>> both
>>>>>>>> these
>>>>>>>>>>>>>>>>> projects
>>>>>>>>>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is
>> there
>>>> a
>>>>>>>> reason to
>>>>>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
>>>>>> invaluable
>>>>>>>> to
>>>>>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
>>>>>>>> themselves. We
>>>>>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
>>>>>> were
>>>>>>>> known
>>>>>>>>>>>>>>>>>>>> failures. I think one of the reasons we have more
>> failing
>>>>>>>> tests now is
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
>>>>>>>> know). This
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> particularly true now because several of the failing
>> tests
>>>>>>>> involve
>>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
>>>>>>>> Currently,
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS
>> issues
>>>>>>>> that are in
>>>>>>>>>>>>>>>>>>> patch available state. Shouldn't we focus on getting
>> these
>>>>>>>> tests fixed
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed
>>>> (applies
>>>>>> to
>>>>>>>> HDFS as
>>>>>>>>>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> Nige
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Todd Lipcon
>>>>>>> Software Engineer, Cloudera
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>
>>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: Patch testing
Posted by Todd Lipcon <to...@cloudera.com>.
On Fri, Jan 7, 2011 at 2:11 PM, Nigel Daley <nd...@mac.com> wrote:
> Hrm, the MR precommit test I'm running has hung (been running for 14 hours
> so far). FWIW, 2 HDFS precommit tests are hung too. I suspect it could be
> the NFS mounts on the machines. I forced a thread dump which you can see in
> the console:
> https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/10/console
>
>
Strange, haven't seen a hang like that before in handleConnectionFailure. It
should retry for 15 minutes max in that loop.
> Any other ideas why these might be hanging?
>
>
There is an HDFS bug right now that can cause hangs on some tests -
HDFS-1529 - would appreciate if someone can take a look. But I don't think
this is responsible for the MR hang above.
-Todd
> On Jan 5, 2011, at 5:42 PM, Todd Lipcon wrote:
>
> > On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <nd...@mac.com> wrote:
> >
> >> Thanks for looking into it Todd. Let's first see if you think it can be
> >> fixed quickly. Let me know.
> >>
> >>
> > No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which
> fixes
> > this test timeout for me.
> >
> > -Todd
> >
> >
> >> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
> >>
> >>> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <nd...@mac.com> wrote:
> >>>
> >>>> Todd, would love to get
> >>>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first
> since
> >>>> this is failing every night on trunk.
> >>>>
> >>>
> >>> What if we disable that test, move that issue to 0.22 blocker, and then
> >>> enable the test-patch? I'll also look into that one today, but if it's
> >>> something that will take a while to fix, I don't think we should hold
> off
> >>> the useful testing for all the other patches.
> >>>
> >>> -Todd
> >>>
> >>> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
> >>>>
> >>>>> Hi Nigel,
> >>>>>
> >>>>> MAPREDUCE-2172 has been fixed for a while. Are there any other
> >> particular
> >>>>> JIRAs you think need to be fixed before the MR test-patch queue gets
> >>>>> enabled? I have a lot of outstanding patches and doing all the
> >> test-patch
> >>>>> turnaround manually on 3 different boxes is a real headache.
> >>>>>
> >>>>> Thanks
> >>>>> -Todd
> >>>>>
> >>>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <nd...@mac.com> wrote:
> >>>>>
> >>>>>> Ok, HDFS is now enabled. You'll see a stream of updates shortly on
> >> the
> >>>> ~30
> >>>>>> Patch Available HDFS issues.
> >>>>>>
> >>>>>> Nige
> >>>>>>
> >>>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
> >>>>>>
> >>>>>>> I committed HDFS-1511 this morning. We should be good to go. I
> can
> >>>>>>> haz snooty robot butler?
> >>>>>>>
> >>>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <
> cos@apache.org>
> >>>>>> wrote:
> >>>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
> >>>>>>>> unless it is done earlier.
> >>>>>>>> --
> >>>>>>>> Take care,
> >>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com>
> >> wrote:
> >>>>>>>>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants
> >> to
> >>>>>>>>> whip one up tonight.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com>
> >> wrote:
> >>>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done
> I'll
> >>>>>> enable hdfs patch testing.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Nige
> >>>>>>>>>>
> >>>>>>>>>> Sent from my iPhone4
> >>>>>>>>>>
> >>>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org
> >
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> One more issue needs to be addressed before test-patch is
> turned
> >> on
> >>>>>> HDFS is
> >>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
> >>>>>>>>>>> --
> >>>>>>>>>>> Take care,
> >>>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
> >> cos@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>>> Considering that because of these 4 faulty cases every patch
> >> will
> >>>> be
> >>>>>>>>>>>> -1'ed a patch author will still have to look at it and make a
> >>>>>> comment
> >>>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
> >>>>>> messier
> >>>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better
> >> way.
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Take care,
> >>>>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com
> >
> >>>>>> wrote:
> >>>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
> >>>>>>>>>>>>>> nothing but dozens of -1'ed patches.
> >>>>>>>>>>>>> There aren't dozens of patches being submitted currently.
> The
> >> -1
> >>>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
> >>>> running
> >>>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson
> does
> >> so
> >>>>>> that
> >>>>>>>>>>>>> the developer doesn't have to.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
> >>>>>> dhruba@gmail.com> wrote:
> >>>>>>>>>>>>>> +1, thanks for doing this.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
> >> jghoman@gmail.com
> >>>>>
> >>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> So, with test-patch updated to show the failing tests,
> saving
> >>>> the
> >>>>>>>>>>>>>>> developers the need to go and verify that the failed tests
> >> are
> >>>>>> all
> >>>>>>>>>>>>>>> known, how do people feel about turning on test-patch again
> >> for
> >>>>>> HDFS
> >>>>>>>>>>>>>>> and mapred? I think it'll help prevent any more tests from
> >>>>>> entering
> >>>>>>>>>>>>>>> the "yeah, we know" category.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> jg
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
> >>>>>> jhoman@yahoo-inc.com> wrote:
> >>>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests
> would
> >>>> need
> >>>>>> to be
> >>>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
> >> Hudson
> >>>>>> could list
> >>>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA). But
> >>>> that's
> >>>>>> still
> >>>>>>>>>>>>>>> quite
> >>>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
> >>>> tests
> >>>>>> and
> >>>>>>>>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are
> a
> >>>> lot
> >>>>>> of
> >>>>>>>>>>>>>>> patches
> >>>>>>>>>>>>>>>> up in the air and several developers are juggling multiple
> >>>>>> patches. The
> >>>>>>>>>>>>>>>> more automation we can have, even if it's not perfect,
> will
> >>>>>> decrease
> >>>>>>>>>>>>>>> errors
> >>>>>>>>>>>>>>>> we may make.
> >>>>>>>>>>>>>>>> -jg
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Nigel Daley wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we
> won't
> >>>>>> turn it on
> >>>>>>>>>>>>>>>>>>> until these projects build and test cleanly. Looks
> like
> >>>> both
> >>>>>> these
> >>>>>>>>>>>>>>> projects
> >>>>>>>>>>>>>>>>>>> currently have test failures.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is
> there
> >> a
> >>>>>> reason to
> >>>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
> >>>> invaluable
> >>>>>> to
> >>>>>>>>>>>>>>> developers
> >>>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
> >>>>>> themselves. We
> >>>>>>>>>>>>>>> didn't
> >>>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
> >>>> were
> >>>>>> known
> >>>>>>>>>>>>>>>>>> failures. I think one of the reasons we have more
> failing
> >>>>>> tests now is
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
> >>>>>> know). This
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> particularly true now because several of the failing
> tests
> >>>>>> involve
> >>>>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
> >>>>>> Currently,
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS
> issues
> >>>>>> that are in
> >>>>>>>>>>>>>>>>> patch available state. Shouldn't we focus on getting
> these
> >>>>>> tests fixed
> >>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed
> >> (applies
> >>>> to
> >>>>>> HDFS as
> >>>>>>>>>>>>>>>>> well) before I turn this on.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>> Nige
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Todd Lipcon
> >>>>> Software Engineer, Cloudera
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>
> >>
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
Hrm, the MR precommit test I'm running has hung (been running for 14 hours so far). FWIW, 2 HDFS precommit tests are hung too. I suspect it could be the NFS mounts on the machines. I forced a thread dump which you can see in the console: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/10/console
Any other ideas why these might be hanging?
Thanks,
Nige
On Jan 5, 2011, at 5:42 PM, Todd Lipcon wrote:
> On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <nd...@mac.com> wrote:
>
>> Thanks for looking into it Todd. Let's first see if you think it can be
>> fixed quickly. Let me know.
>>
>>
> No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which fixes
> this test timeout for me.
>
> -Todd
>
>
>> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
>>
>>> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <nd...@mac.com> wrote:
>>>
>>>> Todd, would love to get
>>>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since
>>>> this is failing every night on trunk.
>>>>
>>>
>>> What if we disable that test, move that issue to 0.22 blocker, and then
>>> enable the test-patch? I'll also look into that one today, but if it's
>>> something that will take a while to fix, I don't think we should hold off
>>> the useful testing for all the other patches.
>>>
>>> -Todd
>>>
>>> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
>>>>
>>>>> Hi Nigel,
>>>>>
>>>>> MAPREDUCE-2172 has been fixed for a while. Are there any other
>> particular
>>>>> JIRAs you think need to be fixed before the MR test-patch queue gets
>>>>> enabled? I have a lot of outstanding patches and doing all the
>> test-patch
>>>>> turnaround manually on 3 different boxes is a real headache.
>>>>>
>>>>> Thanks
>>>>> -Todd
>>>>>
>>>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <nd...@mac.com> wrote:
>>>>>
>>>>>> Ok, HDFS is now enabled. You'll see a stream of updates shortly on
>> the
>>>> ~30
>>>>>> Patch Available HDFS issues.
>>>>>>
>>>>>> Nige
>>>>>>
>>>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>>>>>>
>>>>>>> I committed HDFS-1511 this morning. We should be good to go. I can
>>>>>>> haz snooty robot butler?
>>>>>>>
>>>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <co...@apache.org>
>>>>>> wrote:
>>>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>>>>>>>> unless it is done earlier.
>>>>>>>> --
>>>>>>>> Take care,
>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com>
>> wrote:
>>>>>>>>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants
>> to
>>>>>>>>> whip one up tonight.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com>
>> wrote:
>>>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
>>>>>> enable hdfs patch testing.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Nige
>>>>>>>>>>
>>>>>>>>>> Sent from my iPhone4
>>>>>>>>>>
>>>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> One more issue needs to be addressed before test-patch is turned
>> on
>>>>>> HDFS is
>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>>>>>>>> --
>>>>>>>>>>> Take care,
>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
>> cos@apache.org>
>>>>>> wrote:
>>>>>>>>>>>> Considering that because of these 4 faulty cases every patch
>> will
>>>> be
>>>>>>>>>>>> -1'ed a patch author will still have to look at it and make a
>>>>>> comment
>>>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
>>>>>> messier
>>>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better
>> way.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Take care,
>>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>>>>>>>> There aren't dozens of patches being submitted currently. The
>> -1
>>>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
>>>> running
>>>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does
>> so
>>>>>> that
>>>>>>>>>>>>> the developer doesn't have to.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
>>>>>> dhruba@gmail.com> wrote:
>>>>>>>>>>>>>> +1, thanks for doing this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
>> jghoman@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, with test-patch updated to show the failing tests, saving
>>>> the
>>>>>>>>>>>>>>> developers the need to go and verify that the failed tests
>> are
>>>>>> all
>>>>>>>>>>>>>>> known, how do people feel about turning on test-patch again
>> for
>>>>>> HDFS
>>>>>>>>>>>>>>> and mapred? I think it'll help prevent any more tests from
>>>>>> entering
>>>>>>>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> jg
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
>>>>>> jhoman@yahoo-inc.com> wrote:
>>>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests would
>>>> need
>>>>>> to be
>>>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
>> Hudson
>>>>>> could list
>>>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA). But
>>>> that's
>>>>>> still
>>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
>>>> tests
>>>>>> and
>>>>>>>>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are a
>>>> lot
>>>>>> of
>>>>>>>>>>>>>>> patches
>>>>>>>>>>>>>>>> up in the air and several developers are juggling multiple
>>>>>> patches. The
>>>>>>>>>>>>>>>> more automation we can have, even if it's not perfect, will
>>>>>> decrease
>>>>>>>>>>>>>>> errors
>>>>>>>>>>>>>>>> we may make.
>>>>>>>>>>>>>>>> -jg
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
>>>>>> turn it on
>>>>>>>>>>>>>>>>>>> until these projects build and test cleanly. Looks like
>>>> both
>>>>>> these
>>>>>>>>>>>>>>> projects
>>>>>>>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is there
>> a
>>>>>> reason to
>>>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
>>>> invaluable
>>>>>> to
>>>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
>>>>>> themselves. We
>>>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
>>>> were
>>>>>> known
>>>>>>>>>>>>>>>>>> failures. I think one of the reasons we have more failing
>>>>>> tests now is
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
>>>>>> know). This
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> particularly true now because several of the failing tests
>>>>>> involve
>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
>>>>>> Currently,
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
>>>>>> that are in
>>>>>>>>>>>>>>>>> patch available state. Shouldn't we focus on getting these
>>>>>> tests fixed
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed
>> (applies
>>>> to
>>>>>> HDFS as
>>>>>>>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Nige
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>
>>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: Patch testing
Posted by Todd Lipcon <to...@cloudera.com>.
On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <nd...@mac.com> wrote:
> Thanks for looking into it Todd. Let's first see if you think it can be
> fixed quickly. Let me know.
>
>
No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which fixes
this test timeout for me.
-Todd
> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
>
> > On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <nd...@mac.com> wrote:
> >
> >> Todd, would love to get
> >> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since
> >> this is failing every night on trunk.
> >>
> >
> > What if we disable that test, move that issue to 0.22 blocker, and then
> > enable the test-patch? I'll also look into that one today, but if it's
> > something that will take a while to fix, I don't think we should hold off
> > the useful testing for all the other patches.
> >
> > -Todd
> >
> > On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
> >>
> >>> Hi Nigel,
> >>>
> >>> MAPREDUCE-2172 has been fixed for a while. Are there any other
> particular
> >>> JIRAs you think need to be fixed before the MR test-patch queue gets
> >>> enabled? I have a lot of outstanding patches and doing all the
> test-patch
> >>> turnaround manually on 3 different boxes is a real headache.
> >>>
> >>> Thanks
> >>> -Todd
> >>>
> >>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <nd...@mac.com> wrote:
> >>>
> >>>> Ok, HDFS is now enabled. You'll see a stream of updates shortly on
> the
> >> ~30
> >>>> Patch Available HDFS issues.
> >>>>
> >>>> Nige
> >>>>
> >>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
> >>>>
> >>>>> I committed HDFS-1511 this morning. We should be good to go. I can
> >>>>> haz snooty robot butler?
> >>>>>
> >>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <co...@apache.org>
> >>>> wrote:
> >>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
> >>>>>> unless it is done earlier.
> >>>>>> --
> >>>>>> Take care,
> >>>>>> Konstantin (Cos) Boudnik
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com>
> wrote:
> >>>>>>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants
> to
> >>>>>>> whip one up tonight.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com>
> wrote:
> >>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
> >>>> enable hdfs patch testing.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Nige
> >>>>>>>>
> >>>>>>>> Sent from my iPhone4
> >>>>>>>>
> >>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> One more issue needs to be addressed before test-patch is turned
> on
> >>>> HDFS is
> >>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
> >>>>>>>>> --
> >>>>>>>>> Take care,
> >>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
> cos@apache.org>
> >>>> wrote:
> >>>>>>>>>> Considering that because of these 4 faulty cases every patch
> will
> >> be
> >>>>>>>>>> -1'ed a patch author will still have to look at it and make a
> >>>> comment
> >>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
> >>>> messier
> >>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better
> way.
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Take care,
> >>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com>
> >>>> wrote:
> >>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
> >>>>>>>>>>>> nothing but dozens of -1'ed patches.
> >>>>>>>>>>> There aren't dozens of patches being submitted currently. The
> -1
> >>>>>>>>>>> isn't the important thing, it's the grunt work of actually
> >> running
> >>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does
> so
> >>>> that
> >>>>>>>>>>> the developer doesn't have to.
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
> >>>> dhruba@gmail.com> wrote:
> >>>>>>>>>>>> +1, thanks for doing this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
> jghoman@gmail.com
> >>>
> >>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> So, with test-patch updated to show the failing tests, saving
> >> the
> >>>>>>>>>>>>> developers the need to go and verify that the failed tests
> are
> >>>> all
> >>>>>>>>>>>>> known, how do people feel about turning on test-patch again
> for
> >>>> HDFS
> >>>>>>>>>>>>> and mapred? I think it'll help prevent any more tests from
> >>>> entering
> >>>>>>>>>>>>> the "yeah, we know" category.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> jg
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
> >>>> jhoman@yahoo-inc.com> wrote:
> >>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests would
> >> need
> >>>> to be
> >>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
> Hudson
> >>>> could list
> >>>>>>>>>>>>>> which tests failed in the message it posts to JIRA). But
> >> that's
> >>>> still
> >>>>>>>>>>>>> quite
> >>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
> >> tests
> >>>> and
> >>>>>>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are a
> >> lot
> >>>> of
> >>>>>>>>>>>>> patches
> >>>>>>>>>>>>>> up in the air and several developers are juggling multiple
> >>>> patches. The
> >>>>>>>>>>>>>> more automation we can have, even if it's not perfect, will
> >>>> decrease
> >>>>>>>>>>>>> errors
> >>>>>>>>>>>>>> we may make.
> >>>>>>>>>>>>>> -jg
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Nigel Daley wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
> >>>> turn it on
> >>>>>>>>>>>>>>>>> until these projects build and test cleanly. Looks like
> >> both
> >>>> these
> >>>>>>>>>>>>> projects
> >>>>>>>>>>>>>>>>> currently have test failures.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is there
> a
> >>>> reason to
> >>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
> >> invaluable
> >>>> to
> >>>>>>>>>>>>> developers
> >>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
> >>>> themselves. We
> >>>>>>>>>>>>> didn't
> >>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
> >> were
> >>>> known
> >>>>>>>>>>>>>>>> failures. I think one of the reasons we have more failing
> >>>> tests now is
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
> >>>> know). This
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> particularly true now because several of the failing tests
> >>>> involve
> >>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
> >>>> Currently,
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
> >>>> that are in
> >>>>>>>>>>>>>>> patch available state. Shouldn't we focus on getting these
> >>>> tests fixed
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed
> (applies
> >> to
> >>>> HDFS as
> >>>>>>>>>>>>>>> well) before I turn this on.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>> Nige
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>
> >>
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
Thanks for looking into it Todd. Let's first see if you think it can be fixed quickly. Let me know.
Thanks,
Nige
On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <nd...@mac.com> wrote:
>
>> Todd, would love to get
>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since
>> this is failing every night on trunk.
>>
>
> What if we disable that test, move that issue to 0.22 blocker, and then
> enable the test-patch? I'll also look into that one today, but if it's
> something that will take a while to fix, I don't think we should hold off
> the useful testing for all the other patches.
>
> -Todd
>
> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
>>
>>> Hi Nigel,
>>>
>>> MAPREDUCE-2172 has been fixed for a while. Are there any other particular
>>> JIRAs you think need to be fixed before the MR test-patch queue gets
>>> enabled? I have a lot of outstanding patches and doing all the test-patch
>>> turnaround manually on 3 different boxes is a real headache.
>>>
>>> Thanks
>>> -Todd
>>>
>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <nd...@mac.com> wrote:
>>>
>>>> Ok, HDFS is now enabled. You'll see a stream of updates shortly on the
>> ~30
>>>> Patch Available HDFS issues.
>>>>
>>>> Nige
>>>>
>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>>>>
>>>>> I committed HDFS-1511 this morning. We should be good to go. I can
>>>>> haz snooty robot butler?
>>>>>
>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <co...@apache.org>
>>>> wrote:
>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>>>>>> unless it is done earlier.
>>>>>> --
>>>>>> Take care,
>>>>>> Konstantin (Cos) Boudnik
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com> wrote:
>>>>>>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to
>>>>>>> whip one up tonight.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com> wrote:
>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
>>>> enable hdfs patch testing.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Nige
>>>>>>>>
>>>>>>>> Sent from my iPhone4
>>>>>>>>
>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org>
>>>> wrote:
>>>>>>>>
>>>>>>>>> One more issue needs to be addressed before test-patch is turned on
>>>> HDFS is
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>>>>>> --
>>>>>>>>> Take care,
>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <co...@apache.org>
>>>> wrote:
>>>>>>>>>> Considering that because of these 4 faulty cases every patch will
>> be
>>>>>>>>>> -1'ed a patch author will still have to look at it and make a
>>>> comment
>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
>>>> messier
>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Take care,
>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com>
>>>> wrote:
>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>>>>>> There aren't dozens of patches being submitted currently. The -1
>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
>> running
>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so
>>>> that
>>>>>>>>>>> the developer doesn't have to.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
>>>> dhruba@gmail.com> wrote:
>>>>>>>>>>>> +1, thanks for doing this.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com
>>>
>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> So, with test-patch updated to show the failing tests, saving
>> the
>>>>>>>>>>>>> developers the need to go and verify that the failed tests are
>>>> all
>>>>>>>>>>>>> known, how do people feel about turning on test-patch again for
>>>> HDFS
>>>>>>>>>>>>> and mapred? I think it'll help prevent any more tests from
>>>> entering
>>>>>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> jg
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
>>>> jhoman@yahoo-inc.com> wrote:
>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests would
>> need
>>>> to be
>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson
>>>> could list
>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA). But
>> that's
>>>> still
>>>>>>>>>>>>> quite
>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
>> tests
>>>> and
>>>>>>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are a
>> lot
>>>> of
>>>>>>>>>>>>> patches
>>>>>>>>>>>>>> up in the air and several developers are juggling multiple
>>>> patches. The
>>>>>>>>>>>>>> more automation we can have, even if it's not perfect, will
>>>> decrease
>>>>>>>>>>>>> errors
>>>>>>>>>>>>>> we may make.
>>>>>>>>>>>>>> -jg
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
>>>> turn it on
>>>>>>>>>>>>>>>>> until these projects build and test cleanly. Looks like
>> both
>>>> these
>>>>>>>>>>>>> projects
>>>>>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is there a
>>>> reason to
>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
>> invaluable
>>>> to
>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
>>>> themselves. We
>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
>> were
>>>> known
>>>>>>>>>>>>>>>> failures. I think one of the reasons we have more failing
>>>> tests now is
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
>>>> know). This
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> particularly true now because several of the failing tests
>>>> involve
>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
>>>> Currently,
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
>>>> that are in
>>>>>>>>>>>>>>> patch available state. Shouldn't we focus on getting these
>>>> tests fixed
>>>>>>>>>>>>> or
>>>>>>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies
>> to
>>>> HDFS as
>>>>>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Nige
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: Patch testing
Posted by Todd Lipcon <to...@cloudera.com>.
On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <nd...@mac.com> wrote:
> Todd, would love to get
> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since
> this is failing every night on trunk.
>
What if we disable that test, move that issue to 0.22 blocker, and then
enable the test-patch? I'll also look into that one today, but if it's
something that will take a while to fix, I don't think we should hold off
the useful testing for all the other patches.
-Todd
On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
>
> > Hi Nigel,
> >
> > MAPREDUCE-2172 has been fixed for a while. Are there any other particular
> > JIRAs you think need to be fixed before the MR test-patch queue gets
> > enabled? I have a lot of outstanding patches and doing all the test-patch
> > turnaround manually on 3 different boxes is a real headache.
> >
> > Thanks
> > -Todd
> >
> > On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <nd...@mac.com> wrote:
> >
> >> Ok, HDFS is now enabled. You'll see a stream of updates shortly on the
> ~30
> >> Patch Available HDFS issues.
> >>
> >> Nige
> >>
> >> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
> >>
> >>> I committed HDFS-1511 this morning. We should be good to go. I can
> >>> haz snooty robot butler?
> >>>
> >>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <co...@apache.org>
> >> wrote:
> >>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
> >>>> unless it is done earlier.
> >>>> --
> >>>> Take care,
> >>>> Konstantin (Cos) Boudnik
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com> wrote:
> >>>>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to
> >>>>> whip one up tonight.
> >>>>>
> >>>>>
> >>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com> wrote:
> >>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
> >> enable hdfs patch testing.
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Nige
> >>>>>>
> >>>>>> Sent from my iPhone4
> >>>>>>
> >>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org>
> >> wrote:
> >>>>>>
> >>>>>>> One more issue needs to be addressed before test-patch is turned on
> >> HDFS is
> >>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
> >>>>>>> --
> >>>>>>> Take care,
> >>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <co...@apache.org>
> >> wrote:
> >>>>>>>> Considering that because of these 4 faulty cases every patch will
> be
> >>>>>>>> -1'ed a patch author will still have to look at it and make a
> >> comment
> >>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
> >> messier
> >>>>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Take care,
> >>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com>
> >> wrote:
> >>>>>>>>>> If HDFS is added to the test-patch queue right now we get
> >>>>>>>>>> nothing but dozens of -1'ed patches.
> >>>>>>>>> There aren't dozens of patches being submitted currently. The -1
> >>>>>>>>> isn't the important thing, it's the grunt work of actually
> running
> >>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so
> >> that
> >>>>>>>>> the developer doesn't have to.
> >>>>>>>>>
> >>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
> >> dhruba@gmail.com> wrote:
> >>>>>>>>>> +1, thanks for doing this.
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com
> >
> >> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> So, with test-patch updated to show the failing tests, saving
> the
> >>>>>>>>>>> developers the need to go and verify that the failed tests are
> >> all
> >>>>>>>>>>> known, how do people feel about turning on test-patch again for
> >> HDFS
> >>>>>>>>>>> and mapred? I think it'll help prevent any more tests from
> >> entering
> >>>>>>>>>>> the "yeah, we know" category.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> jg
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
> >> jhoman@yahoo-inc.com> wrote:
> >>>>>>>>>>>> True, each patch would get a -1 and the failing tests would
> need
> >> to be
> >>>>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson
> >> could list
> >>>>>>>>>>>> which tests failed in the message it posts to JIRA). But
> that's
> >> still
> >>>>>>>>>>> quite
> >>>>>>>>>>>> a bit less error-prone work than if the developer runs the
> tests
> >> and
> >>>>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are a
> lot
> >> of
> >>>>>>>>>>> patches
> >>>>>>>>>>>> up in the air and several developers are juggling multiple
> >> patches. The
> >>>>>>>>>>>> more automation we can have, even if it's not perfect, will
> >> decrease
> >>>>>>>>>>> errors
> >>>>>>>>>>>> we may make.
> >>>>>>>>>>>> -jg
> >>>>>>>>>>>>
> >>>>>>>>>>>> Nigel Daley wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
> >> turn it on
> >>>>>>>>>>>>>>> until these projects build and test cleanly. Looks like
> both
> >> these
> >>>>>>>>>>> projects
> >>>>>>>>>>>>>>> currently have test failures.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Assuming the projects are compiling and building, is there a
> >> reason to
> >>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
> invaluable
> >> to
> >>>>>>>>>>> developers
> >>>>>>>>>>>>>> who then don't have to run the tests and test-patch
> >> themselves. We
> >>>>>>>>>>> didn't
> >>>>>>>>>>>>>> turn Hudson off when it was working previously and there
> were
> >> known
> >>>>>>>>>>>>>> failures. I think one of the reasons we have more failing
> >> tests now is
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
> >> know). This
> >>>>>>>>>>> is
> >>>>>>>>>>>>>> particularly true now because several of the failing tests
> >> involve
> >>>>>>>>>>> tests
> >>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
> >> Currently,
> >>>>>>>>>>> that
> >>>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
> >> that are in
> >>>>>>>>>>>>> patch available state. Shouldn't we focus on getting these
> >> tests fixed
> >>>>>>>>>>> or
> >>>>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies
> to
> >> HDFS as
> >>>>>>>>>>>>> well) before I turn this on.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>> Nige
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
Todd, would love to get https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since this is failing every night on trunk.
Nige
On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
> Hi Nigel,
>
> MAPREDUCE-2172 has been fixed for a while. Are there any other particular
> JIRAs you think need to be fixed before the MR test-patch queue gets
> enabled? I have a lot of outstanding patches and doing all the test-patch
> turnaround manually on 3 different boxes is a real headache.
>
> Thanks
> -Todd
>
> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <nd...@mac.com> wrote:
>
>> Ok, HDFS is now enabled. You'll see a stream of updates shortly on the ~30
>> Patch Available HDFS issues.
>>
>> Nige
>>
>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>>
>>> I committed HDFS-1511 this morning. We should be good to go. I can
>>> haz snooty robot butler?
>>>
>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>>>> unless it is done earlier.
>>>> --
>>>> Take care,
>>>> Konstantin (Cos) Boudnik
>>>>
>>>>
>>>>
>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com> wrote:
>>>>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to
>>>>> whip one up tonight.
>>>>>
>>>>>
>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com> wrote:
>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
>> enable hdfs patch testing.
>>>>>>
>>>>>> Cheers,
>>>>>> Nige
>>>>>>
>>>>>> Sent from my iPhone4
>>>>>>
>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>>>>>>
>>>>>>> One more issue needs to be addressed before test-patch is turned on
>> HDFS is
>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>>>> --
>>>>>>> Take care,
>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <co...@apache.org>
>> wrote:
>>>>>>>> Considering that because of these 4 faulty cases every patch will be
>>>>>>>> -1'ed a patch author will still have to look at it and make a
>> comment
>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
>> messier
>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Take care,
>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com>
>> wrote:
>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>>>> There aren't dozens of patches being submitted currently. The -1
>>>>>>>>> isn't the important thing, it's the grunt work of actually running
>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so
>> that
>>>>>>>>> the developer doesn't have to.
>>>>>>>>>
>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
>> dhruba@gmail.com> wrote:
>>>>>>>>>> +1, thanks for doing this.
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com>
>> wrote:
>>>>>>>>>>
>>>>>>>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>>>>>>>> developers the need to go and verify that the failed tests are
>> all
>>>>>>>>>>> known, how do people feel about turning on test-patch again for
>> HDFS
>>>>>>>>>>> and mapred? I think it'll help prevent any more tests from
>> entering
>>>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> jg
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
>> jhoman@yahoo-inc.com> wrote:
>>>>>>>>>>>> True, each patch would get a -1 and the failing tests would need
>> to be
>>>>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson
>> could list
>>>>>>>>>>>> which tests failed in the message it posts to JIRA). But that's
>> still
>>>>>>>>>>> quite
>>>>>>>>>>>> a bit less error-prone work than if the developer runs the tests
>> and
>>>>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are a lot
>> of
>>>>>>>>>>> patches
>>>>>>>>>>>> up in the air and several developers are juggling multiple
>> patches. The
>>>>>>>>>>>> more automation we can have, even if it's not perfect, will
>> decrease
>>>>>>>>>>> errors
>>>>>>>>>>>> we may make.
>>>>>>>>>>>> -jg
>>>>>>>>>>>>
>>>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
>> turn it on
>>>>>>>>>>>>>>> until these projects build and test cleanly. Looks like both
>> these
>>>>>>>>>>> projects
>>>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Assuming the projects are compiling and building, is there a
>> reason to
>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable
>> to
>>>>>>>>>>> developers
>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
>> themselves. We
>>>>>>>>>>> didn't
>>>>>>>>>>>>>> turn Hudson off when it was working previously and there were
>> known
>>>>>>>>>>>>>> failures. I think one of the reasons we have more failing
>> tests now is
>>>>>>>>>>> the
>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
>> know). This
>>>>>>>>>>> is
>>>>>>>>>>>>>> particularly true now because several of the failing tests
>> involve
>>>>>>>>>>> tests
>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
>> Currently,
>>>>>>>>>>> that
>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
>> that are in
>>>>>>>>>>>>> patch available state. Shouldn't we focus on getting these
>> tests fixed
>>>>>>>>>>> or
>>>>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to
>> HDFS as
>>>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Nige
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: Patch testing
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Nigel,
MAPREDUCE-2172 has been fixed for a while. Are there any other particular
JIRAs you think need to be fixed before the MR test-patch queue gets
enabled? I have a lot of outstanding patches and doing all the test-patch
turnaround manually on 3 different boxes is a real headache.
Thanks
-Todd
On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <nd...@mac.com> wrote:
> Ok, HDFS is now enabled. You'll see a stream of updates shortly on the ~30
> Patch Available HDFS issues.
>
> Nige
>
> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>
> > I committed HDFS-1511 this morning. We should be good to go. I can
> > haz snooty robot butler?
> >
> > On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
> >> unless it is done earlier.
> >> --
> >> Take care,
> >> Konstantin (Cos) Boudnik
> >>
> >>
> >>
> >> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com> wrote:
> >>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to
> >>> whip one up tonight.
> >>>
> >>>
> >>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com> wrote:
> >>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
> enable hdfs patch testing.
> >>>>
> >>>> Cheers,
> >>>> Nige
> >>>>
> >>>> Sent from my iPhone4
> >>>>
> >>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >>>>
> >>>>> One more issue needs to be addressed before test-patch is turned on
> HDFS is
> >>>>> https://issues.apache.org/jira/browse/HDFS-1511
> >>>>> --
> >>>>> Take care,
> >>>>> Konstantin (Cos) Boudnik
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <co...@apache.org>
> wrote:
> >>>>>> Considering that because of these 4 faulty cases every patch will be
> >>>>>> -1'ed a patch author will still have to look at it and make a
> comment
> >>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
> messier
> >>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
> >>>>>>
> >>>>>> --
> >>>>>> Take care,
> >>>>>> Konstantin (Cos) Boudnik
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com>
> wrote:
> >>>>>>>> If HDFS is added to the test-patch queue right now we get
> >>>>>>>> nothing but dozens of -1'ed patches.
> >>>>>>> There aren't dozens of patches being submitted currently. The -1
> >>>>>>> isn't the important thing, it's the grunt work of actually running
> >>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so
> that
> >>>>>>> the developer doesn't have to.
> >>>>>>>
> >>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
> dhruba@gmail.com> wrote:
> >>>>>>>> +1, thanks for doing this.
> >>>>>>>>
> >>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>>> So, with test-patch updated to show the failing tests, saving the
> >>>>>>>>> developers the need to go and verify that the failed tests are
> all
> >>>>>>>>> known, how do people feel about turning on test-patch again for
> HDFS
> >>>>>>>>> and mapred? I think it'll help prevent any more tests from
> entering
> >>>>>>>>> the "yeah, we know" category.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> jg
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
> jhoman@yahoo-inc.com> wrote:
> >>>>>>>>>> True, each patch would get a -1 and the failing tests would need
> to be
> >>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson
> could list
> >>>>>>>>>> which tests failed in the message it posts to JIRA). But that's
> still
> >>>>>>>>> quite
> >>>>>>>>>> a bit less error-prone work than if the developer runs the tests
> and
> >>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are a lot
> of
> >>>>>>>>> patches
> >>>>>>>>>> up in the air and several developers are juggling multiple
> patches. The
> >>>>>>>>>> more automation we can have, even if it's not perfect, will
> decrease
> >>>>>>>>> errors
> >>>>>>>>>> we may make.
> >>>>>>>>>> -jg
> >>>>>>>>>>
> >>>>>>>>>> Nigel Daley wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
> turn it on
> >>>>>>>>>>>>> until these projects build and test cleanly. Looks like both
> these
> >>>>>>>>> projects
> >>>>>>>>>>>>> currently have test failures.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Assuming the projects are compiling and building, is there a
> reason to
> >>>>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable
> to
> >>>>>>>>> developers
> >>>>>>>>>>>> who then don't have to run the tests and test-patch
> themselves. We
> >>>>>>>>> didn't
> >>>>>>>>>>>> turn Hudson off when it was working previously and there were
> known
> >>>>>>>>>>>> failures. I think one of the reasons we have more failing
> tests now is
> >>>>>>>>> the
> >>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
> know). This
> >>>>>>>>> is
> >>>>>>>>>>>> particularly true now because several of the failing tests
> involve
> >>>>>>>>> tests
> >>>>>>>>>>>> timing out, making the whole testing regime even longer.
> >>>>>>>>>>>
> >>>>>>>>>>> Every single patch would get a -1 and need investigation.
> Currently,
> >>>>>>>>> that
> >>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
> that are in
> >>>>>>>>>>> patch available state. Shouldn't we focus on getting these
> tests fixed
> >>>>>>>>> or
> >>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to
> HDFS as
> >>>>>>>>>>> well) before I turn this on.
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers,
> >>>>>>>>>>> Nige
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Connect to me at http://www.facebook.com/dhruba
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
Ok, HDFS is now enabled. You'll see a stream of updates shortly on the ~30 Patch Available HDFS issues.
Nige
On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
> I committed HDFS-1511 this morning. We should be good to go. I can
> haz snooty robot butler?
>
> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <co...@apache.org> wrote:
>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>> unless it is done earlier.
>> --
>> Take care,
>> Konstantin (Cos) Boudnik
>>
>>
>>
>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com> wrote:
>>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to
>>> whip one up tonight.
>>>
>>>
>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com> wrote:
>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enable hdfs patch testing.
>>>>
>>>> Cheers,
>>>> Nige
>>>>
>>>> Sent from my iPhone4
>>>>
>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org> wrote:
>>>>
>>>>> One more issue needs to be addressed before test-patch is turned on HDFS is
>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>> --
>>>>> Take care,
>>>>> Konstantin (Cos) Boudnik
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <co...@apache.org> wrote:
>>>>>> Considering that because of these 4 faulty cases every patch will be
>>>>>> -1'ed a patch author will still have to look at it and make a comment
>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but messier
>>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>>>>
>>>>>> --
>>>>>> Take care,
>>>>>> Konstantin (Cos) Boudnik
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com> wrote:
>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>> There aren't dozens of patches being submitted currently. The -1
>>>>>>> isn't the important thing, it's the grunt work of actually running
>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so that
>>>>>>> the developer doesn't have to.
>>>>>>>
>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
>>>>>>>> +1, thanks for doing this.
>>>>>>>>
>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>>>>>> developers the need to go and verify that the failed tests are all
>>>>>>>>> known, how do people feel about turning on test-patch again for HDFS
>>>>>>>>> and mapred? I think it'll help prevent any more tests from entering
>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> jg
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>>>>>>>>>> True, each patch would get a -1 and the failing tests would need to be
>>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson could list
>>>>>>>>>> which tests failed in the message it posts to JIRA). But that's still
>>>>>>>>> quite
>>>>>>>>>> a bit less error-prone work than if the developer runs the tests and
>>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are a lot of
>>>>>>>>> patches
>>>>>>>>>> up in the air and several developers are juggling multiple patches. The
>>>>>>>>>> more automation we can have, even if it's not perfect, will decrease
>>>>>>>>> errors
>>>>>>>>>> we may make.
>>>>>>>>>> -jg
>>>>>>>>>>
>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>>>>>>>>>> until these projects build and test cleanly. Looks like both these
>>>>>>>>> projects
>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>
>>>>>>>>>>>> Assuming the projects are compiling and building, is there a reason to
>>>>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable to
>>>>>>>>> developers
>>>>>>>>>>>> who then don't have to run the tests and test-patch themselves. We
>>>>>>>>> didn't
>>>>>>>>>>>> turn Hudson off when it was working previously and there were known
>>>>>>>>>>>> failures. I think one of the reasons we have more failing tests now is
>>>>>>>>> the
>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I know). This
>>>>>>>>> is
>>>>>>>>>>>> particularly true now because several of the failing tests involve
>>>>>>>>> tests
>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>
>>>>>>>>>>> Every single patch would get a -1 and need investigation. Currently,
>>>>>>>>> that
>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues that are in
>>>>>>>>>>> patch available state. Shouldn't we focus on getting these tests fixed
>>>>>>>>> or
>>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Nige
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
Re: Patch testing
Posted by Jakob Homan <jg...@gmail.com>.
I committed HDFS-1511 this morning. We should be good to go. I can
haz snooty robot butler?
On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
> unless it is done earlier.
> --
> Take care,
> Konstantin (Cos) Boudnik
>
>
>
> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com> wrote:
>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to
>> whip one up tonight.
>>
>>
>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com> wrote:
>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enable hdfs patch testing.
>>>
>>> Cheers,
>>> Nige
>>>
>>> Sent from my iPhone4
>>>
>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org> wrote:
>>>
>>>> One more issue needs to be addressed before test-patch is turned on HDFS is
>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>> --
>>>> Take care,
>>>> Konstantin (Cos) Boudnik
>>>>
>>>>
>>>>
>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <co...@apache.org> wrote:
>>>>> Considering that because of these 4 faulty cases every patch will be
>>>>> -1'ed a patch author will still have to look at it and make a comment
>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but messier
>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>>>
>>>>> --
>>>>> Take care,
>>>>> Konstantin (Cos) Boudnik
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com> wrote:
>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>> nothing but dozens of -1'ed patches.
>>>>>> There aren't dozens of patches being submitted currently. The -1
>>>>>> isn't the important thing, it's the grunt work of actually running
>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so that
>>>>>> the developer doesn't have to.
>>>>>>
>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
>>>>>>> +1, thanks for doing this.
>>>>>>>
>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
>>>>>>>
>>>>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>>>>> developers the need to go and verify that the failed tests are all
>>>>>>>> known, how do people feel about turning on test-patch again for HDFS
>>>>>>>> and mapred? I think it'll help prevent any more tests from entering
>>>>>>>> the "yeah, we know" category.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> jg
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>>>>>>>>> True, each patch would get a -1 and the failing tests would need to be
>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson could list
>>>>>>>>> which tests failed in the message it posts to JIRA). But that's still
>>>>>>>> quite
>>>>>>>>> a bit less error-prone work than if the developer runs the tests and
>>>>>>>>> test-patch themselves. Also, with 22 being cut, there are a lot of
>>>>>>>> patches
>>>>>>>>> up in the air and several developers are juggling multiple patches. The
>>>>>>>>> more automation we can have, even if it's not perfect, will decrease
>>>>>>>> errors
>>>>>>>>> we may make.
>>>>>>>>> -jg
>>>>>>>>>
>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>
>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>
>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>>>>>>>>> until these projects build and test cleanly. Looks like both these
>>>>>>>> projects
>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>
>>>>>>>>>>> Assuming the projects are compiling and building, is there a reason to
>>>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable to
>>>>>>>> developers
>>>>>>>>>>> who then don't have to run the tests and test-patch themselves. We
>>>>>>>> didn't
>>>>>>>>>>> turn Hudson off when it was working previously and there were known
>>>>>>>>>>> failures. I think one of the reasons we have more failing tests now is
>>>>>>>> the
>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I know). This
>>>>>>>> is
>>>>>>>>>>> particularly true now because several of the failing tests involve
>>>>>>>> tests
>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>
>>>>>>>>>> Every single patch would get a -1 and need investigation. Currently,
>>>>>>>> that
>>>>>>>>>> would be about 83 investigations between MR and HDFS issues that are in
>>>>>>>>>> patch available state. Shouldn't we focus on getting these tests fixed
>>>>>>>> or
>>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Nige
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>
Re: Patch testing
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks Jacob. I am wasted already but I can do it on Sun, I think,
unless it is done earlier.
--
Take care,
Konstantin (Cos) Boudnik
On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jg...@gmail.com> wrote:
> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to
> whip one up tonight.
>
>
> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com> wrote:
>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enable hdfs patch testing.
>>
>> Cheers,
>> Nige
>>
>> Sent from my iPhone4
>>
>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org> wrote:
>>
>>> One more issue needs to be addressed before test-patch is turned on HDFS is
>>> https://issues.apache.org/jira/browse/HDFS-1511
>>> --
>>> Take care,
>>> Konstantin (Cos) Boudnik
>>>
>>>
>>>
>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <co...@apache.org> wrote:
>>>> Considering that because of these 4 faulty cases every patch will be
>>>> -1'ed a patch author will still have to look at it and make a comment
>>>> why this particular -1 isn't valid. Lesser work, perhaps, but messier
>>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>>
>>>> --
>>>> Take care,
>>>> Konstantin (Cos) Boudnik
>>>>
>>>>
>>>>
>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com> wrote:
>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>> nothing but dozens of -1'ed patches.
>>>>> There aren't dozens of patches being submitted currently. The -1
>>>>> isn't the important thing, it's the grunt work of actually running
>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so that
>>>>> the developer doesn't have to.
>>>>>
>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
>>>>>> +1, thanks for doing this.
>>>>>>
>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
>>>>>>
>>>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>>>> developers the need to go and verify that the failed tests are all
>>>>>>> known, how do people feel about turning on test-patch again for HDFS
>>>>>>> and mapred? I think it'll help prevent any more tests from entering
>>>>>>> the "yeah, we know" category.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> jg
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>>>>>>>> True, each patch would get a -1 and the failing tests would need to be
>>>>>>>> verified as those known bad (BTW, it would be great if Hudson could list
>>>>>>>> which tests failed in the message it posts to JIRA). But that's still
>>>>>>> quite
>>>>>>>> a bit less error-prone work than if the developer runs the tests and
>>>>>>>> test-patch themselves. Also, with 22 being cut, there are a lot of
>>>>>>> patches
>>>>>>>> up in the air and several developers are juggling multiple patches. The
>>>>>>>> more automation we can have, even if it's not perfect, will decrease
>>>>>>> errors
>>>>>>>> we may make.
>>>>>>>> -jg
>>>>>>>>
>>>>>>>> Nigel Daley wrote:
>>>>>>>>>
>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>
>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>>>>>>>> until these projects build and test cleanly. Looks like both these
>>>>>>> projects
>>>>>>>>>>> currently have test failures.
>>>>>>>>>>
>>>>>>>>>> Assuming the projects are compiling and building, is there a reason to
>>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable to
>>>>>>> developers
>>>>>>>>>> who then don't have to run the tests and test-patch themselves. We
>>>>>>> didn't
>>>>>>>>>> turn Hudson off when it was working previously and there were known
>>>>>>>>>> failures. I think one of the reasons we have more failing tests now is
>>>>>>> the
>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I know). This
>>>>>>> is
>>>>>>>>>> particularly true now because several of the failing tests involve
>>>>>>> tests
>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>
>>>>>>>>> Every single patch would get a -1 and need investigation. Currently,
>>>>>>> that
>>>>>>>>> would be about 83 investigations between MR and HDFS issues that are in
>>>>>>>>> patch available state. Shouldn't we focus on getting these tests fixed
>>>>>>> or
>>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>>>>>>>> well) before I turn this on.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Nige
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>
>>>>>
>>>>
>>
>
Re: Patch testing
Posted by Jakob Homan <jg...@gmail.com>.
Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to
whip one up tonight.
On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <nd...@mac.com> wrote:
> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enable hdfs patch testing.
>
> Cheers,
> Nige
>
> Sent from my iPhone4
>
> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> One more issue needs to be addressed before test-patch is turned on HDFS is
>> https://issues.apache.org/jira/browse/HDFS-1511
>> --
>> Take care,
>> Konstantin (Cos) Boudnik
>>
>>
>>
>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <co...@apache.org> wrote:
>>> Considering that because of these 4 faulty cases every patch will be
>>> -1'ed a patch author will still have to look at it and make a comment
>>> why this particular -1 isn't valid. Lesser work, perhaps, but messier
>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>
>>> --
>>> Take care,
>>> Konstantin (Cos) Boudnik
>>>
>>>
>>>
>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com> wrote:
>>>>> If HDFS is added to the test-patch queue right now we get
>>>>> nothing but dozens of -1'ed patches.
>>>> There aren't dozens of patches being submitted currently. The -1
>>>> isn't the important thing, it's the grunt work of actually running
>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so that
>>>> the developer doesn't have to.
>>>>
>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
>>>>> +1, thanks for doing this.
>>>>>
>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
>>>>>
>>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>>> developers the need to go and verify that the failed tests are all
>>>>>> known, how do people feel about turning on test-patch again for HDFS
>>>>>> and mapred? I think it'll help prevent any more tests from entering
>>>>>> the "yeah, we know" category.
>>>>>>
>>>>>> Thanks,
>>>>>> jg
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>>>>>>> True, each patch would get a -1 and the failing tests would need to be
>>>>>>> verified as those known bad (BTW, it would be great if Hudson could list
>>>>>>> which tests failed in the message it posts to JIRA). But that's still
>>>>>> quite
>>>>>>> a bit less error-prone work than if the developer runs the tests and
>>>>>>> test-patch themselves. Also, with 22 being cut, there are a lot of
>>>>>> patches
>>>>>>> up in the air and several developers are juggling multiple patches. The
>>>>>>> more automation we can have, even if it's not perfect, will decrease
>>>>>> errors
>>>>>>> we may make.
>>>>>>> -jg
>>>>>>>
>>>>>>> Nigel Daley wrote:
>>>>>>>>
>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>
>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>>>>>>> until these projects build and test cleanly. Looks like both these
>>>>>> projects
>>>>>>>>>> currently have test failures.
>>>>>>>>>
>>>>>>>>> Assuming the projects are compiling and building, is there a reason to
>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable to
>>>>>> developers
>>>>>>>>> who then don't have to run the tests and test-patch themselves. We
>>>>>> didn't
>>>>>>>>> turn Hudson off when it was working previously and there were known
>>>>>>>>> failures. I think one of the reasons we have more failing tests now is
>>>>>> the
>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I know). This
>>>>>> is
>>>>>>>>> particularly true now because several of the failing tests involve
>>>>>> tests
>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>
>>>>>>>> Every single patch would get a -1 and need investigation. Currently,
>>>>>> that
>>>>>>>> would be about 83 investigations between MR and HDFS issues that are in
>>>>>>>> patch available state. Shouldn't we focus on getting these tests fixed
>>>>>> or
>>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>>>>>>> well) before I turn this on.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Nige
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>
>>>>
>>>
>
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enable hdfs patch testing.
Cheers,
Nige
Sent from my iPhone4
On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <co...@apache.org> wrote:
> One more issue needs to be addressed before test-patch is turned on HDFS is
> https://issues.apache.org/jira/browse/HDFS-1511
> --
> Take care,
> Konstantin (Cos) Boudnik
>
>
>
> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <co...@apache.org> wrote:
>> Considering that because of these 4 faulty cases every patch will be
>> -1'ed a patch author will still have to look at it and make a comment
>> why this particular -1 isn't valid. Lesser work, perhaps, but messier
>> IMO. I'm not blocking it - I just feel like there's a better way.
>>
>> --
>> Take care,
>> Konstantin (Cos) Boudnik
>>
>>
>>
>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com> wrote:
>>>> If HDFS is added to the test-patch queue right now we get
>>>> nothing but dozens of -1'ed patches.
>>> There aren't dozens of patches being submitted currently. The -1
>>> isn't the important thing, it's the grunt work of actually running
>>> (and waiting) for the tests, test-patch, etc. that Hudson does so that
>>> the developer doesn't have to.
>>>
>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
>>>> +1, thanks for doing this.
>>>>
>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
>>>>
>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>> developers the need to go and verify that the failed tests are all
>>>>> known, how do people feel about turning on test-patch again for HDFS
>>>>> and mapred? I think it'll help prevent any more tests from entering
>>>>> the "yeah, we know" category.
>>>>>
>>>>> Thanks,
>>>>> jg
>>>>>
>>>>>
>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>>>>>> True, each patch would get a -1 and the failing tests would need to be
>>>>>> verified as those known bad (BTW, it would be great if Hudson could list
>>>>>> which tests failed in the message it posts to JIRA). But that's still
>>>>> quite
>>>>>> a bit less error-prone work than if the developer runs the tests and
>>>>>> test-patch themselves. Also, with 22 being cut, there are a lot of
>>>>> patches
>>>>>> up in the air and several developers are juggling multiple patches. The
>>>>>> more automation we can have, even if it's not perfect, will decrease
>>>>> errors
>>>>>> we may make.
>>>>>> -jg
>>>>>>
>>>>>> Nigel Daley wrote:
>>>>>>>
>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>
>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>>>>>> until these projects build and test cleanly. Looks like both these
>>>>> projects
>>>>>>>>> currently have test failures.
>>>>>>>>
>>>>>>>> Assuming the projects are compiling and building, is there a reason to
>>>>>>>> not turn it on despite the test failures? Hudson is invaluable to
>>>>> developers
>>>>>>>> who then don't have to run the tests and test-patch themselves. We
>>>>> didn't
>>>>>>>> turn Hudson off when it was working previously and there were known
>>>>>>>> failures. I think one of the reasons we have more failing tests now is
>>>>> the
>>>>>>>> higher cost of doing Hudson's work (not a great excuse I know). This
>>>>> is
>>>>>>>> particularly true now because several of the failing tests involve
>>>>> tests
>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>
>>>>>>> Every single patch would get a -1 and need investigation. Currently,
>>>>> that
>>>>>>> would be about 83 investigations between MR and HDFS issues that are in
>>>>>>> patch available state. Shouldn't we focus on getting these tests fixed
>>>>> or
>>>>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>>>>>> well) before I turn this on.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Nige
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Connect to me at http://www.facebook.com/dhruba
>>>>
>>>
>>
Re: Patch testing
Posted by Konstantin Boudnik <co...@apache.org>.
One more issue needs to be addressed before test-patch is turned on HDFS is
https://issues.apache.org/jira/browse/HDFS-1511
--
Take care,
Konstantin (Cos) Boudnik
On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <co...@apache.org> wrote:
> Considering that because of these 4 faulty cases every patch will be
> -1'ed a patch author will still have to look at it and make a comment
> why this particular -1 isn't valid. Lesser work, perhaps, but messier
> IMO. I'm not blocking it - I just feel like there's a better way.
>
> --
> Take care,
> Konstantin (Cos) Boudnik
>
>
>
> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com> wrote:
>>> If HDFS is added to the test-patch queue right now we get
>>> nothing but dozens of -1'ed patches.
>> There aren't dozens of patches being submitted currently. The -1
>> isn't the important thing, it's the grunt work of actually running
>> (and waiting) for the tests, test-patch, etc. that Hudson does so that
>> the developer doesn't have to.
>>
>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
>>> +1, thanks for doing this.
>>>
>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
>>>
>>>> So, with test-patch updated to show the failing tests, saving the
>>>> developers the need to go and verify that the failed tests are all
>>>> known, how do people feel about turning on test-patch again for HDFS
>>>> and mapred? I think it'll help prevent any more tests from entering
>>>> the "yeah, we know" category.
>>>>
>>>> Thanks,
>>>> jg
>>>>
>>>>
>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>>>> > True, each patch would get a -1 and the failing tests would need to be
>>>> > verified as those known bad (BTW, it would be great if Hudson could list
>>>> > which tests failed in the message it posts to JIRA). But that's still
>>>> quite
>>>> > a bit less error-prone work than if the developer runs the tests and
>>>> > test-patch themselves. Also, with 22 being cut, there are a lot of
>>>> patches
>>>> > up in the air and several developers are juggling multiple patches. The
>>>> > more automation we can have, even if it's not perfect, will decrease
>>>> errors
>>>> > we may make.
>>>> > -jg
>>>> >
>>>> > Nigel Daley wrote:
>>>> >>
>>>> >> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>> >>
>>>> >>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>> >>>> until these projects build and test cleanly. Looks like both these
>>>> projects
>>>> >>>> currently have test failures.
>>>> >>>
>>>> >>> Assuming the projects are compiling and building, is there a reason to
>>>> >>> not turn it on despite the test failures? Hudson is invaluable to
>>>> developers
>>>> >>> who then don't have to run the tests and test-patch themselves. We
>>>> didn't
>>>> >>> turn Hudson off when it was working previously and there were known
>>>> >>> failures. I think one of the reasons we have more failing tests now is
>>>> the
>>>> >>> higher cost of doing Hudson's work (not a great excuse I know). This
>>>> is
>>>> >>> particularly true now because several of the failing tests involve
>>>> tests
>>>> >>> timing out, making the whole testing regime even longer.
>>>> >>
>>>> >> Every single patch would get a -1 and need investigation. Currently,
>>>> that
>>>> >> would be about 83 investigations between MR and HDFS issues that are in
>>>> >> patch available state. Shouldn't we focus on getting these tests fixed
>>>> or
>>>> >> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>>> >> well) before I turn this on.
>>>> >>
>>>> >> Cheers,
>>>> >> Nige
>>>> >
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Connect to me at http://www.facebook.com/dhruba
>>>
>>
>
Re: Patch testing
Posted by Konstantin Boudnik <co...@apache.org>.
Considering that because of these 4 faulty cases every patch will be
-1'ed a patch author will still have to look at it and make a comment
why this particular -1 isn't valid. Lesser work, perhaps, but messier
IMO. I'm not blocking it - I just feel like there's a better way.
--
Take care,
Konstantin (Cos) Boudnik
On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jg...@gmail.com> wrote:
>> If HDFS is added to the test-patch queue right now we get
>> nothing but dozens of -1'ed patches.
> There aren't dozens of patches being submitted currently. The -1
> isn't the important thing, it's the grunt work of actually running
> (and waiting) for the tests, test-patch, etc. that Hudson does so that
> the developer doesn't have to.
>
> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
>> +1, thanks for doing this.
>>
>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
>>
>>> So, with test-patch updated to show the failing tests, saving the
>>> developers the need to go and verify that the failed tests are all
>>> known, how do people feel about turning on test-patch again for HDFS
>>> and mapred? I think it'll help prevent any more tests from entering
>>> the "yeah, we know" category.
>>>
>>> Thanks,
>>> jg
>>>
>>>
>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>>> > True, each patch would get a -1 and the failing tests would need to be
>>> > verified as those known bad (BTW, it would be great if Hudson could list
>>> > which tests failed in the message it posts to JIRA). But that's still
>>> quite
>>> > a bit less error-prone work than if the developer runs the tests and
>>> > test-patch themselves. Also, with 22 being cut, there are a lot of
>>> patches
>>> > up in the air and several developers are juggling multiple patches. The
>>> > more automation we can have, even if it's not perfect, will decrease
>>> errors
>>> > we may make.
>>> > -jg
>>> >
>>> > Nigel Daley wrote:
>>> >>
>>> >> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>> >>
>>> >>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>> >>>> until these projects build and test cleanly. Looks like both these
>>> projects
>>> >>>> currently have test failures.
>>> >>>
>>> >>> Assuming the projects are compiling and building, is there a reason to
>>> >>> not turn it on despite the test failures? Hudson is invaluable to
>>> developers
>>> >>> who then don't have to run the tests and test-patch themselves. We
>>> didn't
>>> >>> turn Hudson off when it was working previously and there were known
>>> >>> failures. I think one of the reasons we have more failing tests now is
>>> the
>>> >>> higher cost of doing Hudson's work (not a great excuse I know). This
>>> is
>>> >>> particularly true now because several of the failing tests involve
>>> tests
>>> >>> timing out, making the whole testing regime even longer.
>>> >>
>>> >> Every single patch would get a -1 and need investigation. Currently,
>>> that
>>> >> would be about 83 investigations between MR and HDFS issues that are in
>>> >> patch available state. Shouldn't we focus on getting these tests fixed
>>> or
>>> >> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>> >> well) before I turn this on.
>>> >>
>>> >> Cheers,
>>> >> Nige
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>
Re: Patch testing
Posted by Jakob Homan <jg...@gmail.com>.
> If HDFS is added to the test-patch queue right now we get
> nothing but dozens of -1'ed patches.
There aren't dozens of patches being submitted currently. The -1
isn't the important thing, it's the grunt work of actually running
(and waiting) for the tests, test-patch, etc. that Hudson does so that
the developer doesn't have to.
On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
> +1, thanks for doing this.
>
> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
>
>> So, with test-patch updated to show the failing tests, saving the
>> developers the need to go and verify that the failed tests are all
>> known, how do people feel about turning on test-patch again for HDFS
>> and mapred? I think it'll help prevent any more tests from entering
>> the "yeah, we know" category.
>>
>> Thanks,
>> jg
>>
>>
>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>> > True, each patch would get a -1 and the failing tests would need to be
>> > verified as those known bad (BTW, it would be great if Hudson could list
>> > which tests failed in the message it posts to JIRA). But that's still
>> quite
>> > a bit less error-prone work than if the developer runs the tests and
>> > test-patch themselves. Also, with 22 being cut, there are a lot of
>> patches
>> > up in the air and several developers are juggling multiple patches. The
>> > more automation we can have, even if it's not perfect, will decrease
>> errors
>> > we may make.
>> > -jg
>> >
>> > Nigel Daley wrote:
>> >>
>> >> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>> >>
>> >>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>> >>>> until these projects build and test cleanly. Looks like both these
>> projects
>> >>>> currently have test failures.
>> >>>
>> >>> Assuming the projects are compiling and building, is there a reason to
>> >>> not turn it on despite the test failures? Hudson is invaluable to
>> developers
>> >>> who then don't have to run the tests and test-patch themselves. We
>> didn't
>> >>> turn Hudson off when it was working previously and there were known
>> >>> failures. I think one of the reasons we have more failing tests now is
>> the
>> >>> higher cost of doing Hudson's work (not a great excuse I know). This
>> is
>> >>> particularly true now because several of the failing tests involve
>> tests
>> >>> timing out, making the whole testing regime even longer.
>> >>
>> >> Every single patch would get a -1 and need investigation. Currently,
>> that
>> >> would be about 83 investigations between MR and HDFS issues that are in
>> >> patch available state. Shouldn't we focus on getting these tests fixed
>> or
>> >> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>> >> well) before I turn this on.
>> >>
>> >> Cheers,
>> >> Nige
>> >
>> >
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
Re: Patch testing
Posted by Dhruba Borthakur <dh...@gmail.com>.
+1, thanks for doing this.
On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
> So, with test-patch updated to show the failing tests, saving the
> developers the need to go and verify that the failed tests are all
> known, how do people feel about turning on test-patch again for HDFS
> and mapred? I think it'll help prevent any more tests from entering
> the "yeah, we know" category.
>
> Thanks,
> jg
>
>
> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
> > True, each patch would get a -1 and the failing tests would need to be
> > verified as those known bad (BTW, it would be great if Hudson could list
> > which tests failed in the message it posts to JIRA). But that's still
> quite
> > a bit less error-prone work than if the developer runs the tests and
> > test-patch themselves. Also, with 22 being cut, there are a lot of
> patches
> > up in the air and several developers are juggling multiple patches. The
> > more automation we can have, even if it's not perfect, will decrease
> errors
> > we may make.
> > -jg
> >
> > Nigel Daley wrote:
> >>
> >> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>
> >>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
> >>>> until these projects build and test cleanly. Looks like both these
> projects
> >>>> currently have test failures.
> >>>
> >>> Assuming the projects are compiling and building, is there a reason to
> >>> not turn it on despite the test failures? Hudson is invaluable to
> developers
> >>> who then don't have to run the tests and test-patch themselves. We
> didn't
> >>> turn Hudson off when it was working previously and there were known
> >>> failures. I think one of the reasons we have more failing tests now is
> the
> >>> higher cost of doing Hudson's work (not a great excuse I know). This
> is
> >>> particularly true now because several of the failing tests involve
> tests
> >>> timing out, making the whole testing regime even longer.
> >>
> >> Every single patch would get a -1 and need investigation. Currently,
> that
> >> would be about 83 investigations between MR and HDFS issues that are in
> >> patch available state. Shouldn't we focus on getting these tests fixed
> or
> >> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
> >> well) before I turn this on.
> >>
> >> Cheers,
> >> Nige
> >
> >
>
--
Connect to me at http://www.facebook.com/dhruba
Re: Patch testing
Posted by Jakob Homan <jg...@gmail.com>.
Doh. Forgot to include a shout-out to Nigel for adding the
failing-tests-list to test-patch. Thanks!
On Fri, Dec 17, 2010 at 3:23 PM, Eli Collins <el...@cloudera.com> wrote:
> +1
>
> I think all the known failing tests should block the release as well.
>
> Thanks,
> Eli
>
> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
>> So, with test-patch updated to show the failing tests, saving the
>> developers the need to go and verify that the failed tests are all
>> known, how do people feel about turning on test-patch again for HDFS
>> and mapred? I think it'll help prevent any more tests from entering
>> the "yeah, we know" category.
>>
>> Thanks,
>> jg
>>
>>
>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>>> True, each patch would get a -1 and the failing tests would need to be
>>> verified as those known bad (BTW, it would be great if Hudson could list
>>> which tests failed in the message it posts to JIRA). But that's still quite
>>> a bit less error-prone work than if the developer runs the tests and
>>> test-patch themselves. Also, with 22 being cut, there are a lot of patches
>>> up in the air and several developers are juggling multiple patches. The
>>> more automation we can have, even if it's not perfect, will decrease errors
>>> we may make.
>>> -jg
>>>
>>> Nigel Daley wrote:
>>>>
>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>
>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>>> until these projects build and test cleanly. Looks like both these projects
>>>>>> currently have test failures.
>>>>>
>>>>> Assuming the projects are compiling and building, is there a reason to
>>>>> not turn it on despite the test failures? Hudson is invaluable to developers
>>>>> who then don't have to run the tests and test-patch themselves. We didn't
>>>>> turn Hudson off when it was working previously and there were known
>>>>> failures. I think one of the reasons we have more failing tests now is the
>>>>> higher cost of doing Hudson's work (not a great excuse I know). This is
>>>>> particularly true now because several of the failing tests involve tests
>>>>> timing out, making the whole testing regime even longer.
>>>>
>>>> Every single patch would get a -1 and need investigation. Currently, that
>>>> would be about 83 investigations between MR and HDFS issues that are in
>>>> patch available state. Shouldn't we focus on getting these tests fixed or
>>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>>> well) before I turn this on.
>>>>
>>>> Cheers,
>>>> Nige
>>>
>>>
>>
>
Re: Patch testing
Posted by Eli Collins <el...@cloudera.com>.
+1
I think all the known failing tests should block the release as well.
Thanks,
Eli
On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jg...@gmail.com> wrote:
> So, with test-patch updated to show the failing tests, saving the
> developers the need to go and verify that the failed tests are all
> known, how do people feel about turning on test-patch again for HDFS
> and mapred? I think it'll help prevent any more tests from entering
> the "yeah, we know" category.
>
> Thanks,
> jg
>
>
> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>> True, each patch would get a -1 and the failing tests would need to be
>> verified as those known bad (BTW, it would be great if Hudson could list
>> which tests failed in the message it posts to JIRA). But that's still quite
>> a bit less error-prone work than if the developer runs the tests and
>> test-patch themselves. Also, with 22 being cut, there are a lot of patches
>> up in the air and several developers are juggling multiple patches. The
>> more automation we can have, even if it's not perfect, will decrease errors
>> we may make.
>> -jg
>>
>> Nigel Daley wrote:
>>>
>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>
>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>> until these projects build and test cleanly. Looks like both these projects
>>>>> currently have test failures.
>>>>
>>>> Assuming the projects are compiling and building, is there a reason to
>>>> not turn it on despite the test failures? Hudson is invaluable to developers
>>>> who then don't have to run the tests and test-patch themselves. We didn't
>>>> turn Hudson off when it was working previously and there were known
>>>> failures. I think one of the reasons we have more failing tests now is the
>>>> higher cost of doing Hudson's work (not a great excuse I know). This is
>>>> particularly true now because several of the failing tests involve tests
>>>> timing out, making the whole testing regime even longer.
>>>
>>> Every single patch would get a -1 and need investigation. Currently, that
>>> would be about 83 investigations between MR and HDFS issues that are in
>>> patch available state. Shouldn't we focus on getting these tests fixed or
>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>> well) before I turn this on.
>>>
>>> Cheers,
>>> Nige
>>
>>
>
Re: Patch testing
Posted by Konstantin Boudnik <co...@apache.org>.
I do believe that it makes sense to wait a bit longer before doing
this. If HDFS is added to the test-patch queue right now we get
nothing but dozens of -1'ed patches. Number of failing tests has been
reduced from 16 down to 4. Of those for 1 is a real bug (reviled by
HDFS-903) and other three (all of the same nature) are needed to be
investigated more.
I'm for fixing the trunk first.
On Fri, Dec 17, 2010 at 15:19, Jakob Homan <jg...@gmail.com> wrote:
> So, with test-patch updated to show the failing tests, saving the
> developers the need to go and verify that the failed tests are all
> known, how do people feel about turning on test-patch again for HDFS
> and mapred? I think it'll help prevent any more tests from entering
> the "yeah, we know" category.
>
> Thanks,
> jg
>
>
> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
>> True, each patch would get a -1 and the failing tests would need to be
>> verified as those known bad (BTW, it would be great if Hudson could list
>> which tests failed in the message it posts to JIRA). But that's still quite
>> a bit less error-prone work than if the developer runs the tests and
>> test-patch themselves. Also, with 22 being cut, there are a lot of patches
>> up in the air and several developers are juggling multiple patches. The
>> more automation we can have, even if it's not perfect, will decrease errors
>> we may make.
>> -jg
>>
>> Nigel Daley wrote:
>>>
>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>
>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>> until these projects build and test cleanly. Looks like both these projects
>>>>> currently have test failures.
>>>>
>>>> Assuming the projects are compiling and building, is there a reason to
>>>> not turn it on despite the test failures? Hudson is invaluable to developers
>>>> who then don't have to run the tests and test-patch themselves. We didn't
>>>> turn Hudson off when it was working previously and there were known
>>>> failures. I think one of the reasons we have more failing tests now is the
>>>> higher cost of doing Hudson's work (not a great excuse I know). This is
>>>> particularly true now because several of the failing tests involve tests
>>>> timing out, making the whole testing regime even longer.
>>>
>>> Every single patch would get a -1 and need investigation. Currently, that
>>> would be about 83 investigations between MR and HDFS issues that are in
>>> patch available state. Shouldn't we focus on getting these tests fixed or
>>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>> well) before I turn this on.
>>>
>>> Cheers,
>>> Nige
>>
>>
>
Re: Patch testing
Posted by Jakob Homan <jg...@gmail.com>.
So, with test-patch updated to show the failing tests, saving the
developers the need to go and verify that the failed tests are all
known, how do people feel about turning on test-patch again for HDFS
and mapred? I think it'll help prevent any more tests from entering
the "yeah, we know" category.
Thanks,
jg
On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jh...@yahoo-inc.com> wrote:
> True, each patch would get a -1 and the failing tests would need to be
> verified as those known bad (BTW, it would be great if Hudson could list
> which tests failed in the message it posts to JIRA). But that's still quite
> a bit less error-prone work than if the developer runs the tests and
> test-patch themselves. Also, with 22 being cut, there are a lot of patches
> up in the air and several developers are juggling multiple patches. The
> more automation we can have, even if it's not perfect, will decrease errors
> we may make.
> -jg
>
> Nigel Daley wrote:
>>
>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>
>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>> until these projects build and test cleanly. Looks like both these projects
>>>> currently have test failures.
>>>
>>> Assuming the projects are compiling and building, is there a reason to
>>> not turn it on despite the test failures? Hudson is invaluable to developers
>>> who then don't have to run the tests and test-patch themselves. We didn't
>>> turn Hudson off when it was working previously and there were known
>>> failures. I think one of the reasons we have more failing tests now is the
>>> higher cost of doing Hudson's work (not a great excuse I know). This is
>>> particularly true now because several of the failing tests involve tests
>>> timing out, making the whole testing regime even longer.
>>
>> Every single patch would get a -1 and need investigation. Currently, that
>> would be about 83 investigations between MR and HDFS issues that are in
>> patch available state. Shouldn't we focus on getting these tests fixed or
>> removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>> well) before I turn this on.
>>
>> Cheers,
>> Nige
>
>
Re: Patch testing
Posted by Jakob Homan <jh...@yahoo-inc.com>.
True, each patch would get a -1 and the failing tests would need to be
verified as those known bad (BTW, it would be great if Hudson could list
which tests failed in the message it posts to JIRA). But that's still
quite a bit less error-prone work than if the developer runs the tests
and test-patch themselves. Also, with 22 being cut, there are a lot of
patches up in the air and several developers are juggling multiple
patches. The more automation we can have, even if it's not perfect,
will decrease errors we may make.
-jg
Nigel Daley wrote:
> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>
>>> It's also ready to run on MapReduce and HDFS but we won't turn it on until these projects build and test cleanly. Looks like both these projects currently have test failures.
>> Assuming the projects are compiling and building, is there a reason to not turn it on despite the test failures? Hudson is invaluable to developers who then don't have to run the tests and test-patch themselves. We didn't turn Hudson off when it was working previously and there were known failures. I think one of the reasons we have more failing tests now is the higher cost of doing Hudson's work (not a great excuse I know). This is particularly true now because several of the failing tests involve tests timing out, making the whole testing regime even longer.
>
> Every single patch would get a -1 and need investigation. Currently, that would be about 83 investigations between MR and HDFS issues that are in patch available state. Shouldn't we focus on getting these tests fixed or removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as well) before I turn this on.
>
> Cheers,
> Nige
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> > It's also ready to run on MapReduce and HDFS but we won't turn it on until these projects build and test cleanly. Looks like both these projects currently have test failures.
>
> Assuming the projects are compiling and building, is there a reason to not turn it on despite the test failures? Hudson is invaluable to developers who then don't have to run the tests and test-patch themselves. We didn't turn Hudson off when it was working previously and there were known failures. I think one of the reasons we have more failing tests now is the higher cost of doing Hudson's work (not a great excuse I know). This is particularly true now because several of the failing tests involve tests timing out, making the whole testing regime even longer.
Every single patch would get a -1 and need investigation. Currently, that would be about 83 investigations between MR and HDFS issues that are in patch available state. Shouldn't we focus on getting these tests fixed or removed/? Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as well) before I turn this on.
Cheers,
Nige
Re: Patch testing
Posted by Jakob Homan <jh...@yahoo-inc.com>.
> It's also ready to run on MapReduce and HDFS but we won't turn it on
until these projects build and test cleanly. Looks like both these
projects currently have test failures.
Assuming the projects are compiling and building, is there a reason to
not turn it on despite the test failures? Hudson is invaluable to
developers who then don't have to run the tests and test-patch
themselves. We didn't turn Hudson off when it was working previously
and there were known failures. I think one of the reasons we have more
failing tests now is the higher cost of doing Hudson's work (not a great
excuse I know). This is particularly true now because several of the
failing tests involve tests timing out, making the whole testing regime
even longer.
-Jakob
Re: Patch testing
Posted by Eli Collins <el...@cloudera.com>.
Here are the jiras for hdfs test failures on trunk:
HDFS-1306 TestFileAppend4 fails
HDFS-1503 TestSaveNamespace fails when FI is enabled
HDFS-1206 TestFiHFlush fails intermittently
HDFS-1496 TestStorageRestore is failing after HDFS-903 fix
HDFS-1502 TestBlockRecovery triggers NPE in assert
HDFS-613 TestBalancer and TestBlockTokenWithDFS fail Balancer assert
On Tue, Nov 16, 2010 at 10:46 PM, Nigel Daley <nd...@mac.com> wrote:
> Just to follow up, the pre-commit patch testing is functioning again on Zookeeper and Hadoop Common. It's also ready to run on MapReduce and HDFS but we won't turn it on until these projects build and test cleanly. Looks like both these projects currently have test failures. Are there Jira's to fix these tests?
>
> New instructions for re-trigger testing of a patch are at https://hudson.apache.org/hudson/job/PreCommit-Admin/
>
> Cheers,
> Nige
>
> On Oct 20, 2010, at 1:30 PM, Nigel Daley wrote:
>
>> Thanks Giri.
>>
>>> Longterm solution:
>>> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
>>
>> Ya, that's what I've got mostly working on my machine. Any open Jira's for this work? If not, I'll open one.
>>
>>> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
>>
>> Not sure that's necessary. I think if we change the patch testing model a little, we can keep the current workflow. Also, given some recent advances in Hudson, I think we can reduce all the patch jobs down to 1 admin job covering *all* projects and 1 job per project -- and we should be able to drastically improve utilization of the slaves we have. Let's move this discussion to a ticket. LMK if I need to open one.
>>
>> Cheers,
>> Nige
>>
>> On Oct 20, 2010, at 1:17 PM, Giridharan Kesavan wrote:
>>
>>>
>>> Old times:
>>> The hudson patch-testing admin job used to run on the master machine hudson.zones.apache.org as this machine used to parse the emails from jira and create the patch queue for testing.
>>>
>>> Current situation:
>>> hudson.zones.apache.org machine is no more a master and we have a new machine called aegis.apache.org as the master machine.
>>>
>>> This machine is configured to process the email and create the patch queue. <done>
>>> Though we are able to get the email and create the patch queue on the new aegis.apache.org machine, still we wont be able to trigger patch admin job on the hudson master as this is not allowed in the new setup.
>>>
>>> Infra team doesn't want us to run any builds on the hudson master. Instead they want all the builds to run on the slave machine.
>>>
>>> I got the password-less access for hudson user from hudson slave h1.grid.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>
>>>
>>> The plan here is to read the patch queue on aegis from h1 and schedule builds. <I'm working on getting this to work>
>>>
>>> Longterm solution:
>>> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
>>> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
>>>
>>> Thanks,
>>> Giri
>>>
>>> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
>>>
>>>> Thanks for getting to it, Nigel. This is absolutely useful and allows to keep
>>>> weeds out of the trunk up to a certain degree.
>>>>
>>>> There were at least two slightly different sources of information about what's
>>>> going on with test-patch process. Would you mind to post a brief about the
>>>> situation so comments can be more substantial?
>>>>
>>>> Thanks,
>>>> Cos
>>>>
>>>> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
>>>>> Folks,
>>>>>
>>>>> I'm working to get the pre-commit patch testing running again for HDFS,
>>>>> HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from
>>>>> day-to-day involvement w/ the project over the last 6 months (new job), I
>>>>> want to check in and make sure folks would still find this pre-commit
>>>>> testing useful. Also, happy to hear any suggested improvement. Let me
>>>>> know.
>>>>>
>>>>> Cheers,
>>>>> Nige
>>>
>>
>
>
Re: Patch testing
Posted by Konstantin Boudnik <co...@apache.org>.
On Tue, Nov 16, 2010 at 10:46PM, Nigel Daley wrote:
> Just to follow up, the pre-commit patch testing is functioning again on
> Zookeeper and Hadoop Common. It's also ready to run on MapReduce and HDFS
> but we won't turn it on until these projects build and test cleanly. Looks
> like both these projects currently have test failures. Are there Jira's to
> fix these tests?
In HDFS some of the tests are covered by JIRAs for sure. I haven't checked MR
lately but it should be the same there.
> New instructions for re-trigger testing of a patch are at https://hudson.apache.org/hudson/job/PreCommit-Admin/
>
> Cheers,
> Nige
>
> On Oct 20, 2010, at 1:30 PM, Nigel Daley wrote:
>
> > Thanks Giri.
> >
> >> Longterm solution:
> >> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
> >
> > Ya, that's what I've got mostly working on my machine. Any open Jira's for this work? If not, I'll open one.
> >
> >> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
> >
> > Not sure that's necessary. I think if we change the patch testing model a little, we can keep the current workflow. Also, given some recent advances in Hudson, I think we can reduce all the patch jobs down to 1 admin job covering *all* projects and 1 job per project -- and we should be able to drastically improve utilization of the slaves we have. Let's move this discussion to a ticket. LMK if I need to open one.
> >
> > Cheers,
> > Nige
> >
> > On Oct 20, 2010, at 1:17 PM, Giridharan Kesavan wrote:
> >
> >>
> >> Old times:
> >> The hudson patch-testing admin job used to run on the master machine hudson.zones.apache.org as this machine used to parse the emails from jira and create the patch queue for testing.
> >>
> >> Current situation:
> >> hudson.zones.apache.org machine is no more a master and we have a new machine called aegis.apache.org as the master machine.
> >>
> >> This machine is configured to process the email and create the patch queue. <done>
> >> Though we are able to get the email and create the patch queue on the new aegis.apache.org machine, still we wont be able to trigger patch admin job on the hudson master as this is not allowed in the new setup.
> >>
> >> Infra team doesn't want us to run any builds on the hudson master. Instead they want all the builds to run on the slave machine.
> >>
> >> I got the password-less access for hudson user from hudson slave h1.grid.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>
> >>
> >> The plan here is to read the patch queue on aegis from h1 and schedule builds. <I'm working on getting this to work>
> >>
> >> Longterm solution:
> >> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
> >> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
> >>
> >> Thanks,
> >> Giri
> >>
> >> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
> >>
> >>> Thanks for getting to it, Nigel. This is absolutely useful and allows to keep
> >>> weeds out of the trunk up to a certain degree.
> >>>
> >>> There were at least two slightly different sources of information about what's
> >>> going on with test-patch process. Would you mind to post a brief about the
> >>> situation so comments can be more substantial?
> >>>
> >>> Thanks,
> >>> Cos
> >>>
> >>> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
> >>>> Folks,
> >>>>
> >>>> I'm working to get the pre-commit patch testing running again for HDFS,
> >>>> HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from
> >>>> day-to-day involvement w/ the project over the last 6 months (new job), I
> >>>> want to check in and make sure folks would still find this pre-commit
> >>>> testing useful. Also, happy to hear any suggested improvement. Let me
> >>>> know.
> >>>>
> >>>> Cheers,
> >>>> Nige
> >>
> >
>
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
Just to follow up, the pre-commit patch testing is functioning again on Zookeeper and Hadoop Common. It's also ready to run on MapReduce and HDFS but we won't turn it on until these projects build and test cleanly. Looks like both these projects currently have test failures. Are there Jira's to fix these tests?
New instructions for re-trigger testing of a patch are at https://hudson.apache.org/hudson/job/PreCommit-Admin/
Cheers,
Nige
On Oct 20, 2010, at 1:30 PM, Nigel Daley wrote:
> Thanks Giri.
>
>> Longterm solution:
>> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
>
> Ya, that's what I've got mostly working on my machine. Any open Jira's for this work? If not, I'll open one.
>
>> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
>
> Not sure that's necessary. I think if we change the patch testing model a little, we can keep the current workflow. Also, given some recent advances in Hudson, I think we can reduce all the patch jobs down to 1 admin job covering *all* projects and 1 job per project -- and we should be able to drastically improve utilization of the slaves we have. Let's move this discussion to a ticket. LMK if I need to open one.
>
> Cheers,
> Nige
>
> On Oct 20, 2010, at 1:17 PM, Giridharan Kesavan wrote:
>
>>
>> Old times:
>> The hudson patch-testing admin job used to run on the master machine hudson.zones.apache.org as this machine used to parse the emails from jira and create the patch queue for testing.
>>
>> Current situation:
>> hudson.zones.apache.org machine is no more a master and we have a new machine called aegis.apache.org as the master machine.
>>
>> This machine is configured to process the email and create the patch queue. <done>
>> Though we are able to get the email and create the patch queue on the new aegis.apache.org machine, still we wont be able to trigger patch admin job on the hudson master as this is not allowed in the new setup.
>>
>> Infra team doesn't want us to run any builds on the hudson master. Instead they want all the builds to run on the slave machine.
>>
>> I got the password-less access for hudson user from hudson slave h1.grid.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>
>>
>> The plan here is to read the patch queue on aegis from h1 and schedule builds. <I'm working on getting this to work>
>>
>> Longterm solution:
>> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
>> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
>>
>> Thanks,
>> Giri
>>
>> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
>>
>>> Thanks for getting to it, Nigel. This is absolutely useful and allows to keep
>>> weeds out of the trunk up to a certain degree.
>>>
>>> There were at least two slightly different sources of information about what's
>>> going on with test-patch process. Would you mind to post a brief about the
>>> situation so comments can be more substantial?
>>>
>>> Thanks,
>>> Cos
>>>
>>> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
>>>> Folks,
>>>>
>>>> I'm working to get the pre-commit patch testing running again for HDFS,
>>>> HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from
>>>> day-to-day involvement w/ the project over the last 6 months (new job), I
>>>> want to check in and make sure folks would still find this pre-commit
>>>> testing useful. Also, happy to hear any suggested improvement. Let me
>>>> know.
>>>>
>>>> Cheers,
>>>> Nige
>>
>
Re: Patch testing
Posted by Giridharan Kesavan <gk...@yahoo-inc.com>.
I ve just filed this jira for discussions about patch-testing.
https://issues.apache.org/jira/browse/HADOOP-7003
Thanks,
-Giri
On Oct 20, 2010, at 1:30 PM, Nigel Daley wrote:
> Thanks Giri.
>
>> Longterm solution:
>> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
>
> Ya, that's what I've got mostly working on my machine. Any open Jira's for this work? If not, I'll open one.
>
>> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
>
> Not sure that's necessary. I think if we change the patch testing model a little, we can keep the current workflow. Also, given some recent advances in Hudson, I think we can reduce all the patch jobs down to 1 admin job covering *all* projects and 1 job per project -- and we should be able to drastically improve utilization of the slaves we have. Let's move this discussion to a ticket. LMK if I need to open one.
>
> Cheers,
> Nige
>
> On Oct 20, 2010, at 1:17 PM, Giridharan Kesavan wrote:
>
>>
>> Old times:
>> The hudson patch-testing admin job used to run on the master machine hudson.zones.apache.org as this machine used to parse the emails from jira and create the patch queue for testing.
>>
>> Current situation:
>> hudson.zones.apache.org machine is no more a master and we have a new machine called aegis.apache.org as the master machine.
>>
>> This machine is configured to process the email and create the patch queue. <done>
>> Though we are able to get the email and create the patch queue on the new aegis.apache.org machine, still we wont be able to trigger patch admin job on the hudson master as this is not allowed in the new setup.
>>
>> Infra team doesn't want us to run any builds on the hudson master. Instead they want all the builds to run on the slave machine.
>>
>> I got the password-less access for hudson user from hudson slave h1.grid.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>
>>
>> The plan here is to read the patch queue on aegis from h1 and schedule builds. <I'm working on getting this to work>
>>
>> Longterm solution:
>> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
>> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
>>
>> Thanks,
>> Giri
>>
>> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
>>
>>> Thanks for getting to it, Nigel. This is absolutely useful and allows to keep
>>> weeds out of the trunk up to a certain degree.
>>>
>>> There were at least two slightly different sources of information about what's
>>> going on with test-patch process. Would you mind to post a brief about the
>>> situation so comments can be more substantial?
>>>
>>> Thanks,
>>> Cos
>>>
>>> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
>>>> Folks,
>>>>
>>>> I'm working to get the pre-commit patch testing running again for HDFS,
>>>> HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from
>>>> day-to-day involvement w/ the project over the last 6 months (new job), I
>>>> want to check in and make sure folks would still find this pre-commit
>>>> testing useful. Also, happy to hear any suggested improvement. Let me
>>>> know.
>>>>
>>>> Cheers,
>>>> Nige
>>
>
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
Thanks Giri.
> Longterm solution:
> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
Ya, that's what I've got mostly working on my machine. Any open Jira's for this work? If not, I'll open one.
> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
Not sure that's necessary. I think if we change the patch testing model a little, we can keep the current workflow. Also, given some recent advances in Hudson, I think we can reduce all the patch jobs down to 1 admin job covering *all* projects and 1 job per project -- and we should be able to drastically improve utilization of the slaves we have. Let's move this discussion to a ticket. LMK if I need to open one.
Cheers,
Nige
On Oct 20, 2010, at 1:17 PM, Giridharan Kesavan wrote:
>
> Old times:
> The hudson patch-testing admin job used to run on the master machine hudson.zones.apache.org as this machine used to parse the emails from jira and create the patch queue for testing.
>
> Current situation:
> hudson.zones.apache.org machine is no more a master and we have a new machine called aegis.apache.org as the master machine.
>
> This machine is configured to process the email and create the patch queue. <done>
> Though we are able to get the email and create the patch queue on the new aegis.apache.org machine, still we wont be able to trigger patch admin job on the hudson master as this is not allowed in the new setup.
>
> Infra team doesn't want us to run any builds on the hudson master. Instead they want all the builds to run on the slave machine.
>
> I got the password-less access for hudson user from hudson slave h1.grid.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>
>
> The plan here is to read the patch queue on aegis from h1 and schedule builds. <I'm working on getting this to work>
>
> Longterm solution:
> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
>
> Thanks,
> Giri
>
> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
>
>> Thanks for getting to it, Nigel. This is absolutely useful and allows to keep
>> weeds out of the trunk up to a certain degree.
>>
>> There were at least two slightly different sources of information about what's
>> going on with test-patch process. Would you mind to post a brief about the
>> situation so comments can be more substantial?
>>
>> Thanks,
>> Cos
>>
>> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
>>> Folks,
>>>
>>> I'm working to get the pre-commit patch testing running again for HDFS,
>>> HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from
>>> day-to-day involvement w/ the project over the last 6 months (new job), I
>>> want to check in and make sure folks would still find this pre-commit
>>> testing useful. Also, happy to hear any suggested improvement. Let me
>>> know.
>>>
>>> Cheers,
>>> Nige
>
Re: Patch testing
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks Giri!
I'm totally agree that we need to move away from the email triggering.
And in order to make changes to JIRA's workflow someone needs to work with
INFRA folks or a person with admin access should suffice?
I'm offering my help for either way, so lemme know.
Cos
On Wed, Oct 20, 2010 at 01:17PM, Giridharan Kesavan wrote:
>
> Old times:
> The hudson patch-testing admin job used to run on the master machine
> hudson.zones.apache.org as this machine used to parse the emails from jira
> and create the patch queue for testing.
>
> Current situation:
> hudson.zones.apache.org machine is no more a master and we have a new
> machine called aegis.apache.org as the master machine.
>
> This machine is configured to process the email and create the patch queue. <done>
> Though we are able to get the email and create the patch queue on the new
> aegis.apache.org machine, still we wont be able to trigger patch admin job
> on the hudson master as this is not allowed in the new setup.
>
> Infra team doesn't want us to run any builds on the hudson master. Instead
> they want all the builds to run on the slave machine.
>
> I got the password-less access for hudson user from hudson slave
> h1.grid.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>
>
> The plan here is to read the patch queue on aegis from h1 and schedule
> builds. <I'm working on getting this to work>
>
> Longterm solution:
> email patch process and queue creation is un-reliable. As a long term
> solution we need to use jira-cli command line tool to read patch directly
> from jira. And doing this would require introducing a new workflow in jira.
> Apart from the current status called "patch-available"
>
> Thanks,
> Giri
>
> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
>
> > Thanks for getting to it, Nigel. This is absolutely useful and allows to keep
> > weeds out of the trunk up to a certain degree.
> >
> > There were at least two slightly different sources of information about what's
> > going on with test-patch process. Would you mind to post a brief about the
> > situation so comments can be more substantial?
> >
> > Thanks,
> > Cos
> >
> > On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
> >> Folks,
> >>
> >> I'm working to get the pre-commit patch testing running again for HDFS,
> >> HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from
> >> day-to-day involvement w/ the project over the last 6 months (new job), I
> >> want to check in and make sure folks would still find this pre-commit
> >> testing useful. Also, happy to hear any suggested improvement. Let me
> >> know.
> >>
> >> Cheers,
> >> Nige
>
Re: Patch testing
Posted by Giridharan Kesavan <gk...@yahoo-inc.com>.
Old times:
The hudson patch-testing admin job used to run on the master machine hudson.zones.apache.org as this machine used to parse the emails from jira and create the patch queue for testing.
Current situation:
hudson.zones.apache.org machine is no more a master and we have a new machine called aegis.apache.org as the master machine.
This machine is configured to process the email and create the patch queue. <done>
Though we are able to get the email and create the patch queue on the new aegis.apache.org machine, still we wont be able to trigger patch admin job on the hudson master as this is not allowed in the new setup.
Infra team doesn't want us to run any builds on the hudson master. Instead they want all the builds to run on the slave machine.
I got the password-less access for hudson user from hudson slave h1.grid.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>
The plan here is to read the patch queue on aegis from h1 and schedule builds. <I'm working on getting this to work>
Longterm solution:
email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira.
And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
Thanks,
Giri
On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
> Thanks for getting to it, Nigel. This is absolutely useful and allows to keep
> weeds out of the trunk up to a certain degree.
>
> There were at least two slightly different sources of information about what's
> going on with test-patch process. Would you mind to post a brief about the
> situation so comments can be more substantial?
>
> Thanks,
> Cos
>
> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
>> Folks,
>>
>> I'm working to get the pre-commit patch testing running again for HDFS,
>> HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from
>> day-to-day involvement w/ the project over the last 6 months (new job), I
>> want to check in and make sure folks would still find this pre-commit
>> testing useful. Also, happy to hear any suggested improvement. Let me
>> know.
>>
>> Cheers,
>> Nige
Re: Patch testing
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks for getting to it, Nigel. This is absolutely useful and allows to keep
weeds out of the trunk up to a certain degree.
There were at least two slightly different sources of information about what's
going on with test-patch process. Would you mind to post a brief about the
situation so comments can be more substantial?
Thanks,
Cos
On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
> Folks,
>
> I'm working to get the pre-commit patch testing running again for HDFS,
> HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from
> day-to-day involvement w/ the project over the last 6 months (new job), I
> want to check in and make sure folks would still find this pre-commit
> testing useful. Also, happy to hear any suggested improvement. Let me
> know.
>
> Cheers,
> Nige
Re: Patch testing
Posted by Eli Collins <el...@cloudera.com>.
Yes! Thanks Nigel.
On Wed, Oct 20, 2010 at 12:47 PM, Nigel Daley <nd...@mac.com> wrote:
> Folks,
>
> I'm working to get the pre-commit patch testing running again for HDFS, HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from day-to-day involvement w/ the project over the last 6 months (new job), I want to check in and make sure folks would still find this pre-commit testing useful. Also, happy to hear any suggested improvement. Let me know.
>
> Cheers,
> Nige
>
Re: Patch testing
Posted by Dhruba Borthakur <dh...@gmail.com>.
Awesome Nigel! That will be of real help.
thanks,
dhruba
On Wed, Oct 20, 2010 at 12:47 PM, Nigel Daley <nd...@mac.com> wrote:
> Folks,
>
> I'm working to get the pre-commit patch testing running again for HDFS,
> HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from
> day-to-day involvement w/ the project over the last 6 months (new job), I
> want to check in and make sure folks would still find this pre-commit
> testing useful. Also, happy to hear any suggested improvement. Let me
> know.
>
> Cheers,
> Nige
>
--
Connect to me at http://www.facebook.com/dhruba
Re: Patch testing
Posted by Konstantin Boudnik <co...@apache.org>.
Actually, there's a way to do it, but is isn't simple as you said and require
certain JIRA management discipline like linking JIRAs properly, for instance.
Another veritable here is to submit patches for all involved project at once or
to be able to postpone a patch validation if patch-A isn't available when
patch-B is submitted (project B depends on A, obviously).
After that everything else is real easy: build dependencies, mvn-install
artifacts, build the dependent one with locally installed artifacts, clean mvn
cache after all. The tricky part though is to guarantee that these chained
builds are executed on the same machine. Otherwise, local mvn repo
synchronization will be one huge mess ;)
So, in other words, let's restore the status quo first ;) IMO
Cos
On Wed, Oct 20, 2010 at 04:37PM, Aaron Myers wrote:
> One feature request, which may be difficult to implement, and may not be
> worth it:
>
> It would be nice to somehow support running tests for patches which require
> changes that span core/MR/HDFS. As it stands, I believe an HDFS patch will
> be built and run against a trunk jar, but if this HDFS patch requires some
> not-yet-committed work that's being done in a HADOOP JIRA, the HDFS tests
> will fail.
>
> Obviously, just getting hudson QA running again is a necessary pre-requisite
> for this. :)
>
> Aaron
Re: Patch testing
Posted by Aaron Myers <at...@cloudera.com>.
One feature request, which may be difficult to implement, and may not be
worth it:
It would be nice to somehow support running tests for patches which require
changes that span core/MR/HDFS. As it stands, I believe an HDFS patch will
be built and run against a trunk jar, but if this HDFS patch requires some
not-yet-committed work that's being done in a HADOOP JIRA, the HDFS tests
will fail.
Obviously, just getting hudson QA running again is a necessary pre-requisite
for this. :)
Aaron
Re: Patch testing
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, Jan 05, 2011 at 08:46PM, Nigel Daley wrote:
> Lol. At least now we know how much it's valued. I could get a foot long from subway.
But hurry Nige - pretty soon it's gonna be worthless ;(
> Cheers,
> Nige
>
> On Jan 5, 2011, at 8:11 PM, Stack <st...@duboce.net> wrote:
>
> > I'll give you $5.00 Nigel if you add HBase to the list below.
> > St.Ack
> >
> > On Wed, Oct 20, 2010 at 1:31 PM, Nigel Daley <nd...@mac.com> wrote:
> >> I think ZK, PIG, etc will also be included in the update I'm working on.
> >>
> >> Cheers,
> >> Nige
> >>
> >> On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote:
> >>
> >>> Great.
> >>>
> >>> Nigel,
> >>> Can you please document in somewhere on how you fixed it? We¹d like to fix
> >>> it for ZooKeeper as well.
> >>>
> >>> Thanks
> >>> mahadev
> >>>
> >>>
> >>> On 10/20/10 1:23 PM, "Owen O'Malley" <om...@apache.org> wrote:
> >>>
> >>>>
> >>>>
> >>>> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
> >>>>
> >>>>> I'm working to get the pre-commit patch testing running again for
> >>>>> HDFS, HADOOP, and MAPREDUCE patches.
> >>>>
> >>>> That would be great, Nigel.
> >>>>
> >>>> Thanks,
> >>>> Owen
> >>>>
> >>>
> >>
> >>
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
Lol. At least now we know how much it's valued. I could get a foot long from subway.
Cheers,
Nige
On Jan 5, 2011, at 8:11 PM, Stack <st...@duboce.net> wrote:
> I'll give you $5.00 Nigel if you add HBase to the list below.
> St.Ack
>
> On Wed, Oct 20, 2010 at 1:31 PM, Nigel Daley <nd...@mac.com> wrote:
>> I think ZK, PIG, etc will also be included in the update I'm working on.
>>
>> Cheers,
>> Nige
>>
>> On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote:
>>
>>> Great.
>>>
>>> Nigel,
>>> Can you please document in somewhere on how you fixed it? We¹d like to fix
>>> it for ZooKeeper as well.
>>>
>>> Thanks
>>> mahadev
>>>
>>>
>>> On 10/20/10 1:23 PM, "Owen O'Malley" <om...@apache.org> wrote:
>>>
>>>>
>>>>
>>>> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
>>>>
>>>>> I'm working to get the pre-commit patch testing running again for
>>>>> HDFS, HADOOP, and MAPREDUCE patches.
>>>>
>>>> That would be great, Nigel.
>>>>
>>>> Thanks,
>>>> Owen
>>>>
>>>
>>
>>
Re: Patch testing
Posted by Stack <st...@duboce.net>.
I'll give you $5.00 Nigel if you add HBase to the list below.
St.Ack
On Wed, Oct 20, 2010 at 1:31 PM, Nigel Daley <nd...@mac.com> wrote:
> I think ZK, PIG, etc will also be included in the update I'm working on.
>
> Cheers,
> Nige
>
> On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote:
>
>> Great.
>>
>> Nigel,
>> Can you please document in somewhere on how you fixed it? We¹d like to fix
>> it for ZooKeeper as well.
>>
>> Thanks
>> mahadev
>>
>>
>> On 10/20/10 1:23 PM, "Owen O'Malley" <om...@apache.org> wrote:
>>
>>>
>>>
>>> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
>>>
>>>> I'm working to get the pre-commit patch testing running again for
>>>> HDFS, HADOOP, and MAPREDUCE patches.
>>>
>>> That would be great, Nigel.
>>>
>>> Thanks,
>>> Owen
>>>
>>
>
>
Re: Patch testing
Posted by Nigel Daley <nd...@mac.com>.
I think ZK, PIG, etc will also be included in the update I'm working on.
Cheers,
Nige
On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote:
> Great.
>
> Nigel,
> Can you please document in somewhere on how you fixed it? We¹d like to fix
> it for ZooKeeper as well.
>
> Thanks
> mahadev
>
>
> On 10/20/10 1:23 PM, "Owen O'Malley" <om...@apache.org> wrote:
>
>>
>>
>> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
>>
>>> I'm working to get the pre-commit patch testing running again for
>>> HDFS, HADOOP, and MAPREDUCE patches.
>>
>> That would be great, Nigel.
>>
>> Thanks,
>> Owen
>>
>
Re: Patch testing
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Great.
Nigel,
Can you please document in somewhere on how you fixed it? We¹d like to fix
it for ZooKeeper as well.
Thanks
mahadev
On 10/20/10 1:23 PM, "Owen O'Malley" <om...@apache.org> wrote:
>
>
> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
>
>> I'm working to get the pre-commit patch testing running again for
>> HDFS, HADOOP, and MAPREDUCE patches.
>
> That would be great, Nigel.
>
> Thanks,
> Owen
>
Re: Patch testing
Posted by Owen O'Malley <om...@apache.org>.
On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
> I'm working to get the pre-commit patch testing running again for
> HDFS, HADOOP, and MAPREDUCE patches.
That would be great, Nigel.
Thanks,
Owen
Re: Patch testing
Posted by Aaron Myers <at...@cloudera.com>.
+1
There have been several tests failing on trunk for entirely too long, and I
believe this will help prevent further regressions while we address those
issues.
Thanks a lot for volunteering to do this work, Nigel.
Aaron
On Wed, Oct 20, 2010 at 3:47 PM, Nigel Daley <nd...@mac.com> wrote:
> Folks,
>
> I'm working to get the pre-commit patch testing running again for HDFS,
> HADOOP, and MAPREDUCE patches. Since I've been on a bit of a hiatus from
> day-to-day involvement w/ the project over the last 6 months (new job), I
> want to check in and make sure folks would still find this pre-commit
> testing useful. Also, happy to hear any suggested improvement. Let me
> know.
>
> Cheers,
> Nige
>