You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Aaron T. Myers" <at...@cloudera.com> on 2012/03/01 01:00:41 UTC
Re: Merging the HA branch to trunk - Wednesday, February 29th
Hello HDFS devs,
The four JIRAs I mentioned below have all now been committed to the HA
branch. We've been running through the test plans posted on HDFS-1623 for
the last week.
The only thing that's been discovered in the interim on the branch that
should perhaps be considered a blocker for the merge is the performance
issues Todd identified. I am of the opinion that we should proceed with the
merge to trunk anyway, despite the presence of these performance
regressions. Todd's done some good work on addressing those, which should
be committed in the next few days, so their presence on trunk should be
brief.
If folks are amenable to the above, I'd like to do the merge to trunk
tomorrow, since it's getting a little late in the day today.
Since this code change is a merge from a branch, we need three +1s to do
it. Clearly I'm +1.
Thanks a lot,
Aaron
--
Aaron T. Myers
Software Engineer, Cloudera
On Wed, Feb 22, 2012 at 6:24 PM, Aaron T. Myers <at...@cloudera.com> wrote:
> Hello HDFS devs,
>
> Work has largely stabilized on the HA-branch in the last few weeks. At
> this point the HA NN project is nearly feature-complete for manual
> failover. We've been running the full test suite nightly, and all automated
> tests have been passing, except for one known test failure which should be
> fixed shortly.
>
> I'd like to begin the process of merging this branch back to HDFS trunk.
> There are still several outstanding sub-JIRAs under the HDFS-1623 and
> HADOOP-7454 umbrella JIRAs, but most of these are either nice-to-haves or
> relate to supporting automatic failover. Once the branch is merged to
> trunk, work on these JIRAs can continue there.
>
> I've identified the following JIRAs which I think should be the only
> remaining blockers for merging to trunk:
>
> HDFS-2904 - Client support for getting delegation tokens in an HA cluster
> HDFS-2920 - Fix remaining TODOs in the code from HA. Mostly little cleanup
> stuff.
> HDFS-2958 - Sweep for remaining proxy construction which doesn't go
> through failover path
> HDFS-2979 - Balancer should use logical URI for creating failover proxy
> (will fix the only current test failure)
>
> All of these JIRAs should be fixed in the next few days.
>
> I propose that, unless more blocker issues are discovered in the interim,
> we merge this branch to trunk one week from today, i.e. Wednesday, February
> 29th. During this time we will also execute the test plans described in the
> test documents attached to HDFS-1623 to try to identify any regressions or
> performance issues in the branch. If you plan to review the code changes or
> the test plan, I ask that you please do so as soon as possible.
>
> Feedback is certainly welcome on this plan.
>
> Thanks a lot,
> Aaron
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>
>
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by "Aaron T. Myers" <at...@cloudera.com>.
On Thu, Mar 1, 2012 at 2:39 PM, Aaron T. Myers <at...@cloudera.com> wrote:
> On Thu, Mar 1, 2012 at 2:10 PM, Tsz Wo Sze <sz...@yahoo.com> wrote:
>
>> - We have not been able to get a stable build on the HA branch since Feb
>> 13. The builds #93 and #94 both failed earlier today. In contrast, the
>> today's build for trunk is stable. It indicates that there are some
>> problems specific to the HA branch.
>>
>
> Note that the last two builds (#95 and #94) failed because of flaky tests
> - one is tracked by HDFS-3037, the other is TesFileAppend4, which has long
> been flaky. Both of these are also known to be flaky on trunk and other
> branches.
>
We've now had several consecutive successful test runs of the HA branch on
Jenkins. Todd and I also addressed all of the findbugs and javadoc warnings
on the branch.
>
>
>> - I also agree with Vinod that the email was not clear about voting. We
>> should put [VOTE] in the subject.
>>
>
> Agreed. I was going to send out another email shortly to hdfs-dev@.
>
I've just started another thread with the subject "[VOTE] Merge the HA
branch to trunk tomorrow - March 2nd."
Thanks,
Aaron
--
Aaron T. Myers
Software Engineer, Cloudera
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by "Aaron T. Myers" <at...@cloudera.com>.
On Thu, Mar 1, 2012 at 2:10 PM, Tsz Wo Sze <sz...@yahoo.com> wrote:
> - We have not been able to get a stable build on the HA branch since Feb
> 13. The builds #93 and #94 both failed earlier today. In contrast, the
> today's build for trunk is stable. It indicates that there are some
> problems specific to the HA branch.
>
Note that the last two builds (#95 and #94) failed because of flaky tests -
one is tracked by HDFS-3037, the other is TesFileAppend4, which has long
been flaky. Both of these are also known to be flaky on trunk and other
branches.
> - I also agree with Vinod that the email was not clear about voting. We
> should put [VOTE] in the subject.
>
Agreed. I was going to send out another email shortly to hdfs-dev@.
--
Aaron T. Myers
Software Engineer, Cloudera
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by "Aaron T. Myers" <at...@cloudera.com>.
On Thu, Mar 1, 2012 at 2:24 PM, Tsz Wo Sze <sz...@yahoo.com> wrote:
> I forgot to mention that we should also check the findbugs warnings, javac
> warnings, javadoc warnings and the build HA branch since there were patches
> committed without Jerkins' +1s. It should not have more warnings than
> trunk or break the build. Otherwise, we should justify why the warnings
> are unavoidable.
Just filed: https://issues.apache.org/jira/browse/HDFS-3039
Note that the reason those patches were committed without Jenkins's +1 is
precisely because test-patch presently can't handle testing against a
non-trunk branch. This deficiency is tracked by the following JIRA:
https://issues.apache.org/jira/browse/HADOOP-7435
Thanks,
Aaron
--
Aaron T. Myers
Software Engineer, Cloudera
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by Tsz Wo Sze <sz...@yahoo.com>.
I forgot to mention that we should also check the findbugs warnings, javac warnings, javadoc warnings and the build HA branch since there were patches committed without Jerkins' +1s. It should not have more warnings than trunk or break the build. Otherwise, we should justify why the warnings are unavoidable.
We did not do a good job earlier. One example is HADOOP-7119. The patch committed introduced findbugs warnigns (HADOOP-7576), javac
warnings (HADOOP-8119), javadoc warnings (HADOOP-7898) and broke mvn
eclipse:eclipse (HADOOP-7567). Although the javac and findbugs warnings were
catched by this Jenkins report, the patch somehow got committed.
Nicholas
________________________________
From: Tsz Wo Sze <sz...@yahoo.com>
To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
Sent: Thursday, March 1, 2012 2:10 PM
Subject: Re: Merging the HA branch to trunk - Wednesday, February 29th
-1
Below are the reasons:
- We have not been able to get a stable build on the HA branch since Feb 13. The builds #93 and #94 both failed earlier today. In contrast, the today's build for trunk is stable. It indicates that there are some problems specific to the HA branch.
- I also agree with Vinod that the email was not clear about voting. We should put [VOTE] in the subject.
Nicholas
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by Tsz Wo Sze <sz...@yahoo.com>.
-1
Below are the reasons:
- We have not been able to get a stable build on the HA branch since Feb 13. The builds #93 and #94 both failed earlier today. In contrast, the today's build for trunk is stable. It indicates that there are some problems specific to the HA branch.
- I also agree with Vinod that the email was not clear about voting. We should put [VOTE] in the subject.
Nicholas
________________________________
From: Vinod Kumar Vavilapalli <vi...@hortonworks.com>
To: hdfs-dev@hadoop.apache.org
Sent: Wednesday, February 29, 2012 7:52 PM
Subject: Re: Merging the HA branch to trunk - Wednesday, February 29th
I think it is better if you start a fresh voting thread. I almost missed
this given originally it was a discussion thread, perhaps others too.
Thanks
+Vinod
On Wed, Feb 29, 2012 at 5:42 PM, Suresh Srinivas <su...@hortonworks.com>wrote:
> I am +1 on merging this to trunk.
>
> On Feb 29, 2012, at 4:03 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > +1 as well.
> >
> > My latest tests after applying the performance fixes indicate that
> > there is no statistically significant performance regression between
> > trunk and HA, even in tests designed to stress test the worst case
> > code paths. The fixes should be committed today or tomorrow, but don't
> > need to hold the merge up.
> >
> > -Todd
> >
> > On Wed, Feb 29, 2012 at 4:00 PM, Aaron T. Myers <at...@cloudera.com>
> wrote:
> >> Hello HDFS devs,
> >>
> >> The four JIRAs I mentioned below have all now been committed to the HA
> >> branch. We've been running through the test plans posted on HDFS-1623
> for
> >> the last week.
> >>
> >> The only thing that's been discovered in the interim on the branch that
> >> should perhaps be considered a blocker for the merge is the performance
> >> issues Todd identified. I am of the opinion that we should proceed with
> the
> >> merge to trunk anyway, despite the presence of these performance
> >> regressions. Todd's done some good work on addressing those, which
> should
> >> be committed in the next few days, so their presence on trunk should be
> >> brief.
> >>
> >> If folks are amenable to the above, I'd like to do the merge to trunk
> >> tomorrow, since it's getting a little late in the day today.
> >>
> >> Since this code change is a merge from a branch, we need three +1s to do
> >> it. Clearly I'm +1.
> >>
> >> Thanks a lot,
> >> Aaron
> >>
> >> --
> >> Aaron T. Myers
> >> Software Engineer, Cloudera
> >>
> >>
> >>
> >> On Wed, Feb 22, 2012 at 6:24 PM, Aaron T. Myers <at...@cloudera.com>
> wrote:
> >>
> >>> Hello HDFS devs,
> >>>
> >>> Work has largely stabilized on the HA-branch in the last few weeks. At
> >>> this point the HA NN project is nearly feature-complete for manual
> >>> failover. We've been running the full test suite nightly, and all
> automated
> >>> tests have been passing, except for one known test failure which
> should be
> >>> fixed shortly.
> >>>
> >>> I'd like to begin the process of merging this branch back to HDFS
> trunk.
> >>> There are still several outstanding sub-JIRAs under the HDFS-1623 and
> >>> HADOOP-7454 umbrella JIRAs, but most of these are either nice-to-haves
> or
> >>> relate to supporting automatic failover. Once the branch is merged to
> >>> trunk, work on these JIRAs can continue there.
> >>>
> >>> I've identified the following JIRAs which I think should be the only
> >>> remaining blockers for merging to trunk:
> >>>
> >>> HDFS-2904 - Client support for getting delegation tokens in an HA
> cluster
> >>> HDFS-2920 - Fix remaining TODOs in the code from HA. Mostly little
> cleanup
> >>> stuff.
> >>> HDFS-2958 - Sweep for remaining proxy construction which doesn't go
> >>> through failover path
> >>> HDFS-2979 - Balancer should use logical URI for creating failover proxy
> >>> (will fix the only current test failure)
> >>>
> >>> All of these JIRAs should be fixed in the next few days.
> >>>
> >>> I propose that, unless more blocker issues are discovered in the
> interim,
> >>> we merge this branch to trunk one week from today, i.e. Wednesday,
> February
> >>> 29th. During this time we will also execute the test plans described
> in the
> >>> test documents attached to HDFS-1623 to try to identify any
> regressions or
> >>> performance issues in the branch. If you plan to review the code
> changes or
> >>> the test plan, I ask that you please do so as soon as possible.
> >>>
> >>> Feedback is certainly welcome on this plan.
> >>>
> >>> Thanks a lot,
> >>> Aaron
> >>>
> >>> --
> >>> Aaron T. Myers
> >>> Software Engineer, Cloudera
> >>>
> >>>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
I think it is better if you start a fresh voting thread. I almost missed
this given originally it was a discussion thread, perhaps others too.
Thanks
+Vinod
On Wed, Feb 29, 2012 at 5:42 PM, Suresh Srinivas <su...@hortonworks.com>wrote:
> I am +1 on merging this to trunk.
>
> On Feb 29, 2012, at 4:03 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > +1 as well.
> >
> > My latest tests after applying the performance fixes indicate that
> > there is no statistically significant performance regression between
> > trunk and HA, even in tests designed to stress test the worst case
> > code paths. The fixes should be committed today or tomorrow, but don't
> > need to hold the merge up.
> >
> > -Todd
> >
> > On Wed, Feb 29, 2012 at 4:00 PM, Aaron T. Myers <at...@cloudera.com>
> wrote:
> >> Hello HDFS devs,
> >>
> >> The four JIRAs I mentioned below have all now been committed to the HA
> >> branch. We've been running through the test plans posted on HDFS-1623
> for
> >> the last week.
> >>
> >> The only thing that's been discovered in the interim on the branch that
> >> should perhaps be considered a blocker for the merge is the performance
> >> issues Todd identified. I am of the opinion that we should proceed with
> the
> >> merge to trunk anyway, despite the presence of these performance
> >> regressions. Todd's done some good work on addressing those, which
> should
> >> be committed in the next few days, so their presence on trunk should be
> >> brief.
> >>
> >> If folks are amenable to the above, I'd like to do the merge to trunk
> >> tomorrow, since it's getting a little late in the day today.
> >>
> >> Since this code change is a merge from a branch, we need three +1s to do
> >> it. Clearly I'm +1.
> >>
> >> Thanks a lot,
> >> Aaron
> >>
> >> --
> >> Aaron T. Myers
> >> Software Engineer, Cloudera
> >>
> >>
> >>
> >> On Wed, Feb 22, 2012 at 6:24 PM, Aaron T. Myers <at...@cloudera.com>
> wrote:
> >>
> >>> Hello HDFS devs,
> >>>
> >>> Work has largely stabilized on the HA-branch in the last few weeks. At
> >>> this point the HA NN project is nearly feature-complete for manual
> >>> failover. We've been running the full test suite nightly, and all
> automated
> >>> tests have been passing, except for one known test failure which
> should be
> >>> fixed shortly.
> >>>
> >>> I'd like to begin the process of merging this branch back to HDFS
> trunk.
> >>> There are still several outstanding sub-JIRAs under the HDFS-1623 and
> >>> HADOOP-7454 umbrella JIRAs, but most of these are either nice-to-haves
> or
> >>> relate to supporting automatic failover. Once the branch is merged to
> >>> trunk, work on these JIRAs can continue there.
> >>>
> >>> I've identified the following JIRAs which I think should be the only
> >>> remaining blockers for merging to trunk:
> >>>
> >>> HDFS-2904 - Client support for getting delegation tokens in an HA
> cluster
> >>> HDFS-2920 - Fix remaining TODOs in the code from HA. Mostly little
> cleanup
> >>> stuff.
> >>> HDFS-2958 - Sweep for remaining proxy construction which doesn't go
> >>> through failover path
> >>> HDFS-2979 - Balancer should use logical URI for creating failover proxy
> >>> (will fix the only current test failure)
> >>>
> >>> All of these JIRAs should be fixed in the next few days.
> >>>
> >>> I propose that, unless more blocker issues are discovered in the
> interim,
> >>> we merge this branch to trunk one week from today, i.e. Wednesday,
> February
> >>> 29th. During this time we will also execute the test plans described
> in the
> >>> test documents attached to HDFS-1623 to try to identify any
> regressions or
> >>> performance issues in the branch. If you plan to review the code
> changes or
> >>> the test plan, I ask that you please do so as soon as possible.
> >>>
> >>> Feedback is certainly welcome on this plan.
> >>>
> >>> Thanks a lot,
> >>> Aaron
> >>>
> >>> --
> >>> Aaron T. Myers
> >>> Software Engineer, Cloudera
> >>>
> >>>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by Suresh Srinivas <su...@hortonworks.com>.
I am +1 on merging this to trunk.
On Feb 29, 2012, at 4:03 PM, Todd Lipcon <to...@cloudera.com> wrote:
> +1 as well.
>
> My latest tests after applying the performance fixes indicate that
> there is no statistically significant performance regression between
> trunk and HA, even in tests designed to stress test the worst case
> code paths. The fixes should be committed today or tomorrow, but don't
> need to hold the merge up.
>
> -Todd
>
> On Wed, Feb 29, 2012 at 4:00 PM, Aaron T. Myers <at...@cloudera.com> wrote:
>> Hello HDFS devs,
>>
>> The four JIRAs I mentioned below have all now been committed to the HA
>> branch. We've been running through the test plans posted on HDFS-1623 for
>> the last week.
>>
>> The only thing that's been discovered in the interim on the branch that
>> should perhaps be considered a blocker for the merge is the performance
>> issues Todd identified. I am of the opinion that we should proceed with the
>> merge to trunk anyway, despite the presence of these performance
>> regressions. Todd's done some good work on addressing those, which should
>> be committed in the next few days, so their presence on trunk should be
>> brief.
>>
>> If folks are amenable to the above, I'd like to do the merge to trunk
>> tomorrow, since it's getting a little late in the day today.
>>
>> Since this code change is a merge from a branch, we need three +1s to do
>> it. Clearly I'm +1.
>>
>> Thanks a lot,
>> Aaron
>>
>> --
>> Aaron T. Myers
>> Software Engineer, Cloudera
>>
>>
>>
>> On Wed, Feb 22, 2012 at 6:24 PM, Aaron T. Myers <at...@cloudera.com> wrote:
>>
>>> Hello HDFS devs,
>>>
>>> Work has largely stabilized on the HA-branch in the last few weeks. At
>>> this point the HA NN project is nearly feature-complete for manual
>>> failover. We've been running the full test suite nightly, and all automated
>>> tests have been passing, except for one known test failure which should be
>>> fixed shortly.
>>>
>>> I'd like to begin the process of merging this branch back to HDFS trunk.
>>> There are still several outstanding sub-JIRAs under the HDFS-1623 and
>>> HADOOP-7454 umbrella JIRAs, but most of these are either nice-to-haves or
>>> relate to supporting automatic failover. Once the branch is merged to
>>> trunk, work on these JIRAs can continue there.
>>>
>>> I've identified the following JIRAs which I think should be the only
>>> remaining blockers for merging to trunk:
>>>
>>> HDFS-2904 - Client support for getting delegation tokens in an HA cluster
>>> HDFS-2920 - Fix remaining TODOs in the code from HA. Mostly little cleanup
>>> stuff.
>>> HDFS-2958 - Sweep for remaining proxy construction which doesn't go
>>> through failover path
>>> HDFS-2979 - Balancer should use logical URI for creating failover proxy
>>> (will fix the only current test failure)
>>>
>>> All of these JIRAs should be fixed in the next few days.
>>>
>>> I propose that, unless more blocker issues are discovered in the interim,
>>> we merge this branch to trunk one week from today, i.e. Wednesday, February
>>> 29th. During this time we will also execute the test plans described in the
>>> test documents attached to HDFS-1623 to try to identify any regressions or
>>> performance issues in the branch. If you plan to review the code changes or
>>> the test plan, I ask that you please do so as soon as possible.
>>>
>>> Feedback is certainly welcome on this plan.
>>>
>>> Thanks a lot,
>>> Aaron
>>>
>>> --
>>> Aaron T. Myers
>>> Software Engineer, Cloudera
>>>
>>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by Todd Lipcon <to...@cloudera.com>.
+1 as well.
My latest tests after applying the performance fixes indicate that
there is no statistically significant performance regression between
trunk and HA, even in tests designed to stress test the worst case
code paths. The fixes should be committed today or tomorrow, but don't
need to hold the merge up.
-Todd
On Wed, Feb 29, 2012 at 4:00 PM, Aaron T. Myers <at...@cloudera.com> wrote:
> Hello HDFS devs,
>
> The four JIRAs I mentioned below have all now been committed to the HA
> branch. We've been running through the test plans posted on HDFS-1623 for
> the last week.
>
> The only thing that's been discovered in the interim on the branch that
> should perhaps be considered a blocker for the merge is the performance
> issues Todd identified. I am of the opinion that we should proceed with the
> merge to trunk anyway, despite the presence of these performance
> regressions. Todd's done some good work on addressing those, which should
> be committed in the next few days, so their presence on trunk should be
> brief.
>
> If folks are amenable to the above, I'd like to do the merge to trunk
> tomorrow, since it's getting a little late in the day today.
>
> Since this code change is a merge from a branch, we need three +1s to do
> it. Clearly I'm +1.
>
> Thanks a lot,
> Aaron
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>
>
>
> On Wed, Feb 22, 2012 at 6:24 PM, Aaron T. Myers <at...@cloudera.com> wrote:
>
>> Hello HDFS devs,
>>
>> Work has largely stabilized on the HA-branch in the last few weeks. At
>> this point the HA NN project is nearly feature-complete for manual
>> failover. We've been running the full test suite nightly, and all automated
>> tests have been passing, except for one known test failure which should be
>> fixed shortly.
>>
>> I'd like to begin the process of merging this branch back to HDFS trunk.
>> There are still several outstanding sub-JIRAs under the HDFS-1623 and
>> HADOOP-7454 umbrella JIRAs, but most of these are either nice-to-haves or
>> relate to supporting automatic failover. Once the branch is merged to
>> trunk, work on these JIRAs can continue there.
>>
>> I've identified the following JIRAs which I think should be the only
>> remaining blockers for merging to trunk:
>>
>> HDFS-2904 - Client support for getting delegation tokens in an HA cluster
>> HDFS-2920 - Fix remaining TODOs in the code from HA. Mostly little cleanup
>> stuff.
>> HDFS-2958 - Sweep for remaining proxy construction which doesn't go
>> through failover path
>> HDFS-2979 - Balancer should use logical URI for creating failover proxy
>> (will fix the only current test failure)
>>
>> All of these JIRAs should be fixed in the next few days.
>>
>> I propose that, unless more blocker issues are discovered in the interim,
>> we merge this branch to trunk one week from today, i.e. Wednesday, February
>> 29th. During this time we will also execute the test plans described in the
>> test documents attached to HDFS-1623 to try to identify any regressions or
>> performance issues in the branch. If you plan to review the code changes or
>> the test plan, I ask that you please do so as soon as possible.
>>
>> Feedback is certainly welcome on this plan.
>>
>> Thanks a lot,
>> Aaron
>>
>> --
>> Aaron T. Myers
>> Software Engineer, Cloudera
>>
>>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by Tsz Wo Sze <sz...@yahoo.com>.
Anyway, could we get a stable build before merging?
Nicholas
________________________________
From: Todd Lipcon <to...@cloudera.com>
To: hdfs-dev@hadoop.apache.org; Tsz Wo Sze <sz...@yahoo.com>
Sent: Wednesday, February 29, 2012 5:04 PM
Subject: Re: Merging the HA branch to trunk - Wednesday, February 29th
On Wed, Feb 29, 2012 at 5:01 PM, Tsz Wo Sze <sz...@yahoo.com> wrote:
> Hi Todd,
>
> Then, why the trunk build did not suffer such problem?
Bad luck? Looks like a network blip - failure to download something
from archive.apache.org.
Todd
>
>
> ________________________________
> From: Todd Lipcon <to...@cloudera.com>
> To: hdfs-dev@hadoop.apache.org; Tsz Wo Sze <sz...@yahoo.com>
> Sent: Wednesday, February 29, 2012 4:36 PM
> Subject: Re: Merging the HA branch to trunk - Wednesday, February 29th
>
> On Wed, Feb 29, 2012 at 4:32 PM, Tsz Wo Sze <sz...@yahoo.com> wrote:
>>
>> Latest HA Build (#92):
>> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-HAbranch-build/92/console
>>
>> [INFO]
>> [INFO] Apache Hadoop HDFS ................................ SUCCESS [8:34.652s]
>> [INFO] Apache Hadoop HttpFS .............................. FAILURE [8.346s]
>> [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
>> [INFO] Apache Hadoop HDFS Project ........................ SKIPPED
>
> The build failure seems to be due to a network issue:
> [get] Getting:
> http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz
> [get] To:
> /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-HAbranch-build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads/tomcat.tar.gz
> ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
> [mkdir] Created dir:
> /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-HAbranch-build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/tomcat.exp
> [exec]
> [exec] gzip: stdin: unexpected end of file
> [exec] tar: Unexpected EOF in archive
> [exec] tar: Unexpected EOF in archive
> [exec] tar: Error is not recoverable: exiting now
>
>
> I was able to successfully run "mvn install" in the hadoop-hdfs-httpfs
> directory on the branch on my machine.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by Todd Lipcon <to...@cloudera.com>.
On Wed, Feb 29, 2012 at 5:01 PM, Tsz Wo Sze <sz...@yahoo.com> wrote:
> Hi Todd,
>
> Then, why the trunk build did not suffer such problem?
Bad luck? Looks like a network blip - failure to download something
from archive.apache.org.
Todd
>
>
> ________________________________
> From: Todd Lipcon <to...@cloudera.com>
> To: hdfs-dev@hadoop.apache.org; Tsz Wo Sze <sz...@yahoo.com>
> Sent: Wednesday, February 29, 2012 4:36 PM
> Subject: Re: Merging the HA branch to trunk - Wednesday, February 29th
>
> On Wed, Feb 29, 2012 at 4:32 PM, Tsz Wo Sze <sz...@yahoo.com> wrote:
>>
>> Latest HA Build (#92):
>> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-HAbranch-build/92/console
>>
>> [INFO]
>> [INFO] Apache Hadoop HDFS ................................ SUCCESS [8:34.652s]
>> [INFO] Apache Hadoop HttpFS .............................. FAILURE [8.346s]
>> [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
>> [INFO] Apache Hadoop HDFS Project ........................ SKIPPED
>
> The build failure seems to be due to a network issue:
> [get] Getting:
> http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz
> [get] To:
> /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-HAbranch-build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads/tomcat.tar.gz
> ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
> [mkdir] Created dir:
> /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-HAbranch-build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/tomcat.exp
> [exec]
> [exec] gzip: stdin: unexpected end of file
> [exec] tar: Unexpected EOF in archive
> [exec] tar: Unexpected EOF in archive
> [exec] tar: Error is not recoverable: exiting now
>
>
> I was able to successfully run "mvn install" in the hadoop-hdfs-httpfs
> directory on the branch on my machine.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by Tsz Wo Sze <sz...@yahoo.com>.
Hi Todd,
Then, why the trunk build did not suffer such problem?
Nicholas
________________________________
From: Todd Lipcon <to...@cloudera.com>
To: hdfs-dev@hadoop.apache.org; Tsz Wo Sze <sz...@yahoo.com>
Sent: Wednesday, February 29, 2012 4:36 PM
Subject: Re: Merging the HA branch to trunk - Wednesday, February 29th
On Wed, Feb 29, 2012 at 4:32 PM, Tsz Wo Sze <sz...@yahoo.com> wrote:
>
> Latest HA Build (#92):
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-HAbranch-build/92/console
>
> [INFO]
> [INFO] Apache Hadoop HDFS ................................ SUCCESS [8:34.652s]
> [INFO] Apache Hadoop HttpFS .............................. FAILURE [8.346s]
> [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
> [INFO] Apache Hadoop HDFS Project ........................ SKIPPED
The build failure seems to be due to a network issue:
[get] Getting:
http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz
[get] To:
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-HAbranch-build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads/tomcat.tar.gz
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
[mkdir] Created dir:
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-HAbranch-build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/tomcat.exp
[exec]
[exec] gzip: stdin: unexpected end of file
[exec] tar: Unexpected EOF in archive
[exec] tar: Unexpected EOF in archive
[exec] tar: Error is not recoverable: exiting now
I was able to successfully run "mvn install" in the hadoop-hdfs-httpfs
directory on the branch on my machine.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by Todd Lipcon <to...@cloudera.com>.
On Wed, Feb 29, 2012 at 4:32 PM, Tsz Wo Sze <sz...@yahoo.com> wrote:
>
> Latest HA Build (#92):
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-HAbranch-build/92/console
>
> [INFO]
> [INFO] Apache Hadoop HDFS ................................ SUCCESS [8:34.652s]
> [INFO] Apache Hadoop HttpFS .............................. FAILURE [8.346s]
> [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
> [INFO] Apache Hadoop HDFS Project ........................ SKIPPED
The build failure seems to be due to a network issue:
[get] Getting:
http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz
[get] To:
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-HAbranch-build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads/tomcat.tar.gz
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
[mkdir] Created dir:
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-HAbranch-build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/tomcat.exp
[exec]
[exec] gzip: stdin: unexpected end of file
[exec] tar: Unexpected EOF in archive
[exec] tar: Unexpected EOF in archive
[exec] tar: Error is not recoverable: exiting now
I was able to successfully run "mvn install" in the hadoop-hdfs-httpfs
directory on the branch on my machine.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Merging the HA branch to trunk - Wednesday, February 29th
Posted by Tsz Wo Sze <sz...@yahoo.com>.
Hi,
The HA build has been unstable since Feb 13. In particular, HttpFS failed in the latest build (#92). On the other hand, the latest trunk HDFS build (#970) is stable. Could we stabilize the HA build before merging?
Nicholas
-------------------------------
Latest HA Build (#92):
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-HAbranch-build/92/console
[INFO]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [8:34.652s]
[INFO] Apache Hadoop HttpFS .............................. FAILURE [8.346s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
[INFO] Apache Hadoop HDFS Project ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ---------------------------------
________________________________
From: Aaron T. Myers <at...@cloudera.com>
To: hdfs-dev@hadoop.apache.org
Sent: Wednesday, February 29, 2012 4:00 PM
Subject: Re: Merging the HA branch to trunk - Wednesday, February 29th
Hello HDFS devs,
The four JIRAs I mentioned below have all now been committed to the HA
branch. We've been running through the test plans posted on HDFS-1623 for
the last week.
The only thing that's been discovered in the interim on the branch that
should perhaps be considered a blocker for the merge is the performance
issues Todd identified. I am of the opinion that we should proceed with the
merge to trunk anyway, despite the presence of these performance
regressions. Todd's done some good work on addressing those, which should
be committed in the next few days, so their presence on trunk should be
brief.
If folks are amenable to the above, I'd like to do the merge to trunk
tomorrow, since it's getting a little late in the day today.
Since this code change is a merge from a branch, we need three +1s to do
it. Clearly I'm +1.
Thanks a lot,
Aaron
--
Aaron T. Myers
Software Engineer, Cloudera
On Wed, Feb 22, 2012 at 6:24 PM, Aaron T. Myers <at...@cloudera.com> wrote:
> Hello HDFS devs,
>
> Work has largely stabilized on the HA-branch in the last few weeks. At
> this point the HA NN project is nearly feature-complete for manual
> failover. We've been running the full test suite nightly, and all automated
> tests have been passing, except for one known test failure which should be
> fixed shortly.
>
> I'd like to begin the process of merging this branch back to HDFS trunk.
> There are still several outstanding sub-JIRAs under the HDFS-1623 and
> HADOOP-7454 umbrella JIRAs, but most of these are either nice-to-haves or
> relate to supporting automatic failover. Once the branch is merged to
> trunk, work on these JIRAs can continue there.
>
> I've identified the following JIRAs which I think should be the only
> remaining blockers for merging to trunk:
>
> HDFS-2904 - Client support for getting delegation tokens in an HA cluster
> HDFS-2920 - Fix remaining TODOs in the code from HA. Mostly little cleanup
> stuff.
> HDFS-2958 - Sweep for remaining proxy construction which doesn't go
> through failover path
> HDFS-2979 - Balancer should use logical URI for creating failover proxy
> (will fix the only current test failure)
>
> All of these JIRAs should be fixed in the next few days.
>
> I propose that, unless more blocker issues are discovered in the interim,
> we merge this branch to trunk one week from today, i.e. Wednesday, February
> 29th. During this time we will also execute the test plans described in the
> test documents attached to HDFS-1623 to try to identify any regressions or
> performance issues in the branch. If you plan to review the code changes or
> the test plan, I ask that you please do so as soon as possible.
>
> Feedback is certainly welcome on this plan.
>
> Thanks a lot,
> Aaron
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>
>