You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jonathan Hsieh <jo...@cloudera.com> on 2013/04/11 18:26:21 UTC
trunk on hadoop2 fixes.
Hey folks,
I plan on committing these patches hanging off this jira late today.
https://issues.apache.org/jira/browse/HBASE-6891
Summary:
* HBASE-7605 (Mini cluster load) and HBASE-7606 (Joined scanners) shows a
completion time regression from h1 to h2 profiles
* HBASE-7636 (3 rs failure recovery) have a problem with hdfs's
shortcircuit reads.
These are quick fixes, to get the hadoop2 build clean. After I get the
build green, I'll pursue root causes.
The failure in HBASE-8324 (HFileOutputFormat failures on yarn/mr2) are the
last consistent failures on my hadoop2 build. I'm still working on it --
yarn/mr2 is a bit of a black box.
Jon.
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: trunk on hadoop2 fixes.
Posted by Ted Yu <yu...@gmail.com>.
I was about to do this validation but got some issue in locating artifacts
off maven repo.
Over in '[VOTE] Release Apache Hadoop 2.0.4-alpha' thread, Andrew reported
compilation error which I confirmed locally.
Will continue with this effort next week.
On Fri, Apr 12, 2013 at 8:28 AM, Stack <st...@duboce.net> wrote:
> On Thu, Apr 11, 2013 at 11:45 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. Share the results on the mailing list if you want.
> >
> > I plan to run HBase test suite after 2.0.4-alpha is released.
> > I think from HBase side, there is no pending issue what would affect the
> > release of 2.0.4-alpha.
> >
>
> You could do us all a favor and try the 2.0.4-alpha candidate before it is
> released (it is kind of useless all around trying it after it has been
> released).
>
> St.Ack
>
Re: trunk on hadoop2 fixes.
Posted by Stack <st...@duboce.net>.
On Thu, Apr 11, 2013 at 11:45 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. Share the results on the mailing list if you want.
>
> I plan to run HBase test suite after 2.0.4-alpha is released.
> I think from HBase side, there is no pending issue what would affect the
> release of 2.0.4-alpha.
>
You could do us all a favor and try the 2.0.4-alpha candidate before it is
released (it is kind of useless all around trying it after it has been
released).
St.Ack
Re: trunk on hadoop2 fixes.
Posted by Ted Yu <yu...@gmail.com>.
bq. Share the results on the mailing list if you want.
I plan to run HBase test suite after 2.0.4-alpha is released.
I think from HBase side, there is no pending issue what would affect the
release of 2.0.4-alpha.
Cheers
On Thu, Apr 11, 2013 at 5:52 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> Ted,
>
> Nothing is stopping you from running the tests against the hadoop's tip.
> Share the results on the mailing list if you want.
>
> Here's my suggestion:
>
> Please don't create a jira and post a patch unless (that triggers the
> precommit build) unless you make it clear that it is speculative work and
> that it won't be immediately committed and would be considered for commit
> only after hadoop 2.0.4 is released.
>
> Jon.
>
> On Thursday, April 11, 2013, Ted Yu wrote:
>
> > Do you mind if I run the test you are fixing against tip of hadoop 2.0
> > branch ?
> >
> > Or, if I understand you correctly, potentially new JIRA would be opened
> on
> > (some of) these tests when 2.0.4-alpha comes out.
> >
> > Cheers
> >
> > On Thu, Apr 11, 2013 at 9:45 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> > > On Thu, Apr 11, 2013 at 9:39 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Jon:
> > > > Thanks for the headsup.
> > > >
> > > > Do you know that you can trigger hadoop QA to run test suite against
> > > hadoop
> > > > 2.0 profile ?
> > > >
> > > > Take a look at:
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/attachment/12575537/7904-v9-hadoop-2.0.txt
> > > >
> > > > Hadoop QA depends on seeing the line below in patch:
> > > >
> > > > hadoop.profile=2.0
> > > >
> > > > Neat trick. We should document this!
> > >
> > >
> > > > You also need some changes to the pom.xml files.
> > > >
> > > > I want to know your strategy w.r.t. re-verifying the fixes when
> > > 2.0.4-alpha
> > > > comes out.
> > > > 2.0.4-alpha RC0 was cut two days ago. RC1 should come by weekend.
> > > >
> > > >
> > > Let's get our current hbase trunk on hadoop2 green most of the time
> > first,
> > > then change to the new version. This way we'll know if it was changes
> to
> > > hbase or changes to hadoop that cause any problems that may occur.
> > >
> > > Jon.
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // Software Engineer, Cloudera
> > > // jon@cloudera.com
> > >
> >
>
Re: trunk on hadoop2 fixes.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Ted,
Nothing is stopping you from running the tests against the hadoop's tip.
Share the results on the mailing list if you want.
Here's my suggestion:
Please don't create a jira and post a patch unless (that triggers the
precommit build) unless you make it clear that it is speculative work and
that it won't be immediately committed and would be considered for commit
only after hadoop 2.0.4 is released.
Jon.
On Thursday, April 11, 2013, Ted Yu wrote:
> Do you mind if I run the test you are fixing against tip of hadoop 2.0
> branch ?
>
> Or, if I understand you correctly, potentially new JIRA would be opened on
> (some of) these tests when 2.0.4-alpha comes out.
>
> Cheers
>
> On Thu, Apr 11, 2013 at 9:45 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > On Thu, Apr 11, 2013 at 9:39 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Jon:
> > > Thanks for the headsup.
> > >
> > > Do you know that you can trigger hadoop QA to run test suite against
> > hadoop
> > > 2.0 profile ?
> > >
> > > Take a look at:
> > >
> > >
> >
> https://issues.apache.org/jira/secure/attachment/12575537/7904-v9-hadoop-2.0.txt
> > >
> > > Hadoop QA depends on seeing the line below in patch:
> > >
> > > hadoop.profile=2.0
> > >
> > > Neat trick. We should document this!
> >
> >
> > > You also need some changes to the pom.xml files.
> > >
> > > I want to know your strategy w.r.t. re-verifying the fixes when
> > 2.0.4-alpha
> > > comes out.
> > > 2.0.4-alpha RC0 was cut two days ago. RC1 should come by weekend.
> > >
> > >
> > Let's get our current hbase trunk on hadoop2 green most of the time
> first,
> > then change to the new version. This way we'll know if it was changes to
> > hbase or changes to hadoop that cause any problems that may occur.
> >
> > Jon.
> >
> > --
> > // Jonathan Hsieh (shay)
> > // Software Engineer, Cloudera
> > // jon@cloudera.com
> >
>
Re: trunk on hadoop2 fixes.
Posted by Ted Yu <yu...@gmail.com>.
Do you mind if I run the test you are fixing against tip of hadoop 2.0
branch ?
Or, if I understand you correctly, potentially new JIRA would be opened on
(some of) these tests when 2.0.4-alpha comes out.
Cheers
On Thu, Apr 11, 2013 at 9:45 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> On Thu, Apr 11, 2013 at 9:39 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Jon:
> > Thanks for the headsup.
> >
> > Do you know that you can trigger hadoop QA to run test suite against
> hadoop
> > 2.0 profile ?
> >
> > Take a look at:
> >
> >
> https://issues.apache.org/jira/secure/attachment/12575537/7904-v9-hadoop-2.0.txt
> >
> > Hadoop QA depends on seeing the line below in patch:
> >
> > hadoop.profile=2.0
> >
> > Neat trick. We should document this!
>
>
> > You also need some changes to the pom.xml files.
> >
> > I want to know your strategy w.r.t. re-verifying the fixes when
> 2.0.4-alpha
> > comes out.
> > 2.0.4-alpha RC0 was cut two days ago. RC1 should come by weekend.
> >
> >
> Let's get our current hbase trunk on hadoop2 green most of the time first,
> then change to the new version. This way we'll know if it was changes to
> hbase or changes to hadoop that cause any problems that may occur.
>
> Jon.
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
Re: trunk on hadoop2 fixes.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
On Thu, Apr 11, 2013 at 9:39 AM, Ted Yu <yu...@gmail.com> wrote:
> Jon:
> Thanks for the headsup.
>
> Do you know that you can trigger hadoop QA to run test suite against hadoop
> 2.0 profile ?
>
> Take a look at:
>
> https://issues.apache.org/jira/secure/attachment/12575537/7904-v9-hadoop-2.0.txt
>
> Hadoop QA depends on seeing the line below in patch:
>
> hadoop.profile=2.0
>
> Neat trick. We should document this!
> You also need some changes to the pom.xml files.
>
> I want to know your strategy w.r.t. re-verifying the fixes when 2.0.4-alpha
> comes out.
> 2.0.4-alpha RC0 was cut two days ago. RC1 should come by weekend.
>
>
Let's get our current hbase trunk on hadoop2 green most of the time first,
then change to the new version. This way we'll know if it was changes to
hbase or changes to hadoop that cause any problems that may occur.
Jon.
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: trunk on hadoop2 fixes.
Posted by Ted Yu <yu...@gmail.com>.
Jon:
Thanks for the headsup.
Do you know that you can trigger hadoop QA to run test suite against hadoop
2.0 profile ?
Take a look at:
https://issues.apache.org/jira/secure/attachment/12575537/7904-v9-hadoop-2.0.txt
Hadoop QA depends on seeing the line below in patch:
hadoop.profile=2.0
You also need some changes to the pom.xml files.
I want to know your strategy w.r.t. re-verifying the fixes when 2.0.4-alpha
comes out.
2.0.4-alpha RC0 was cut two days ago. RC1 should come by weekend.
Cheers
On Thu, Apr 11, 2013 at 9:26 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> Hey folks,
>
> I plan on committing these patches hanging off this jira late today.
> https://issues.apache.org/jira/browse/HBASE-6891
>
> Summary:
> * HBASE-7605 (Mini cluster load) and HBASE-7606 (Joined scanners) shows a
> completion time regression from h1 to h2 profiles
> * HBASE-7636 (3 rs failure recovery) have a problem with hdfs's
> shortcircuit reads.
>
> These are quick fixes, to get the hadoop2 build clean. After I get the
> build green, I'll pursue root causes.
>
> The failure in HBASE-8324 (HFileOutputFormat failures on yarn/mr2) are the
> last consistent failures on my hadoop2 build. I'm still working on it --
> yarn/mr2 is a bit of a black box.
>
> Jon.
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
Re: trunk on hadoop2 fixes.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
On Thursday, April 11, 2013, Ted Yu wrote:
> HBASE-8382 doesn't exist, as of this email. Can you confirm the JIRA number
> ?
my bad . hbase-8324
>
> > > The failure in HBASE-8324 (HFileOutputFormat failures on yarn/mr2) are
> > > the last consistent failures on my hadoop2 build. I'm still working on
> > it
> > > -- yarn/mr2 is a bit of a black box.
> > >
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: trunk on hadoop2 fixes.
Posted by Ted Yu <yu...@gmail.com>.
HBASE-8382 doesn't exist, as of this email. Can you confirm the JIRA number
?
bq. updating the precommit script so that it runs the hadoop 2.0 profile or
some set of it as well?
I think the above is a good idea. With this in place, we can easily find
out about (binary) incompatibility of HBase trunk w.r.t. hadoop 2.0
In the first cut, we can select the tests you recently fixed.
Cheers
On Thu, Apr 11, 2013 at 5:55 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> I believe I got the last consistently failing tests against hadoop2 fixed
> this afternoon (HBASE-8382). If and when this is green, what do folks
> think about updating the precommit script so that it runs the hadoop 2.0
> profile or some set of it as well? (We could probably get away with doing
> just the Medium or just the Large tests).
>
> Thanks,
> Jon.
>
> On Thu, Apr 11, 2013 at 9:26 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > Hey folks,
> >
> > I plan on committing these patches hanging off this jira late today.
> > https://issues.apache.org/jira/browse/HBASE-6891
> >
> > Summary:
> > * HBASE-7605 (Mini cluster load) and HBASE-7606 (Joined scanners) shows a
> > completion time regression from h1 to h2 profiles
> > * HBASE-7636 (3 rs failure recovery) have a problem with hdfs's
> > shortcircuit reads.
> >
> > These are quick fixes, to get the hadoop2 build clean. After I get the
> > build green, I'll pursue root causes.
> >
> > The failure in HBASE-8324 (HFileOutputFormat failures on yarn/mr2) are
> > the last consistent failures on my hadoop2 build. I'm still working on
> it
> > -- yarn/mr2 is a bit of a black box.
> >
> > Jon.
> >
> > --
> > // Jonathan Hsieh (shay)
> > // Software Engineer, Cloudera
> > // jon@cloudera.com
> >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
Re: trunk on hadoop2 fixes.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
I believe I got the last consistently failing tests against hadoop2 fixed
this afternoon (HBASE-8382). If and when this is green, what do folks
think about updating the precommit script so that it runs the hadoop 2.0
profile or some set of it as well? (We could probably get away with doing
just the Medium or just the Large tests).
Thanks,
Jon.
On Thu, Apr 11, 2013 at 9:26 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> Hey folks,
>
> I plan on committing these patches hanging off this jira late today.
> https://issues.apache.org/jira/browse/HBASE-6891
>
> Summary:
> * HBASE-7605 (Mini cluster load) and HBASE-7606 (Joined scanners) shows a
> completion time regression from h1 to h2 profiles
> * HBASE-7636 (3 rs failure recovery) have a problem with hdfs's
> shortcircuit reads.
>
> These are quick fixes, to get the hadoop2 build clean. After I get the
> build green, I'll pursue root causes.
>
> The failure in HBASE-8324 (HFileOutputFormat failures on yarn/mr2) are
> the last consistent failures on my hadoop2 build. I'm still working on it
> -- yarn/mr2 is a bit of a black box.
>
> Jon.
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com