You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Matt Foley <ma...@apache.org> on 2012/07/12 23:26:43 UTC
update re: Hadoop-1.1.0-rc0
The following additional patches are committed to branch-1.1 and will be in
1.1.0:
- HADOOP-8365 - Add flag to disable durable sync
- HADOOP-8399 - Remove JDK5 dependency from Hadoop 1.0+ line
- HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
- HADOOP-8445 - Token should not print the password in toString
- MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job is
executed with 0 maps
- MAPREDUCE-2129 - Job may hang if
mapreduce.job.committer.setup.cleanup.needed=false and
mapreduce.map/reduce.failures.maxpercent>0
I've been trying to build and test the RC0, but am having build problems
with the 32-bit version. Should have it figured out soon.
The following patch is committed to branch-1 and I'll see if I can apply it
to branch-1.1 before my next build:
- HDFS-3617 - Port HDFS-96 to branch-1 (support blocks greater than 2GB)
The following patch is committed to branch-1 but had conflicts with
branch-1.1 and will not be in RC0:
- HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
The following patches are deferred, as they do not have currently committed
patches:
- HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for bzip2)
- MAPREDUCE-4049 - plugin for generic shuffle service
Thanks,
--Matt
On Sat, Jul 7, 2012 at 6:32 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Also, I have updated HADOOP-8417 against 1.1.0 and we need to include it.
> Otherwise, 1.1 will have the same issues for the downstream projects as
> 1.0.3
> had.
>
> Cos
>
> On Sat, Jul 07, 2012 at 05:52PM, Konstantin Boudnik wrote:
> > Matt,
> >
> > Thanks for the update.
> >
> > HADOOP-8399 would be beneficial for BigTop release and it is marked for
> 1.1.0
> > release. The patch is available for a while now and if someone can
> review I'd
> > go ahead and commit it today.
> >
> > I am working on the content of 0.3.1 BigTop release and will shortly
> post the
> > vote for it. Once Hadoop 1.1 rc is cut we'll start testing it with the
> rest of
> > the stack.
> >
> > Cos
> >
> > On Fri, Jul 06, 2012 at 02:24PM, Matt Foley wrote:
> > > Hi Cos,
> > > the query string didn't come thru on the link you sent, but the jira
> query
> > > I use is:
> > > project in (HADOOP,HDFS,MAPREDUCE) and (("Target Version/s" =
> '1.1.0'
> > > and (fixVersion != '1.1.0' or fixVersion is EMPTY)) or (fixVersion =
> > > '1.1.0' and "Target Version/s" is EMPTY)) and (status != Closed and
> status
> > > != Resolved) ORDER BY KEY
> > >
> > > You're correct that there are quite a few, currently 107, open jiras
> > > originally targeted for 1.1.0 that do not have committed fixes. Many
> of
> > > these are just the inherited backlog of previously identified work. I
> need
> > > to move them to "Target Version/s" = 1.1.1.
> > >
> > > Folks have requested that the following currently open jiras be
> included in
> > > 1.1.0:
> > >
> > > HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> > > HADOOP-8445 - Token should not print the password in toString
> > > HDFS-96 - HDFS does not support blocks greater than 2GB
> > > HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
> > > MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job
> is
> > > executed with 0 maps
> > > MAPREDUCE-2129 - Job may hang if
> > > mapreduce.job.committer.setup.cleanup.needed=false and
> > > mapreduce.map/reduce.failures.maxpercent>0
> > > ---
> > > MAPREDUCE-4049 - plugin for generic shuffle service
> > > HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for
> bzip2)
> > >
> > > The first six are simple patches that I am comfortable including.
> > > The last two are complex patches that have not yet been committed.
> > > I am planning to defer those two to 1.1.1.
> > >
> > > Beyond that, I'm going to cut 1.1.0-rc0 from the current state of
> > > branch-1.1.
> > > I'm planning to do that this weekend. This is obviously delayed from
> the
> > > previous plan, for which I apologize.
> > >
> > > Comments welcome.
> > > --Matt
> > >
> > >
> > > On Tue, Jul 3, 2012 at 8:32 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > >
> > > > Hi Matt.
> > > >
> > > > I am picking up the hat of BigTop's maintainer for Hadoop 1.x line.
> And I
> > > > wanted to sync up with about the Hadoop 1.1 release outlook,
> progress,
> > > > what help
> > > > you might need, etc.
> > > >
> > > > I see a few jiras left open in the release
> > > > http://is.gd/OyuaNQ
> > > > Is this the correct representation of the current status?
> > > > How I can help from BigTop side (I haven't yet finalized the stack's
> > > > versions), etc. Looking forward for your input. Thanks.
> > > >
> > > > Cos
> > > >
> > > > On Fri, May 25, 2012 at 02:49PM, Matt Foley wrote:
> > > > > Greetings. With the approval of a public vote on common-dev@, I
> have
> > > > > branched Hadoop branch-1 to create branch-1.1. From this, I will
> create
> > > > a
> > > > > release candidate RC-0 for Hadoop-1.1.0, hopefully to be available
> > > > shortly
> > > > > after this weekend.
> > > > >
> > > > > There are over 80 patches in branch-1, over and above the contents
> of
> > > > > hadoop-1.0.3. So I anticipate that some stabilization will be
> needed,
> > > > > before the RC can be approved as a 1.1.0 release. Your
> participation in
> > > > > assuring a stable RC is very important. When it becomes available,
> > > > please
> > > > > download it and work with it to determine whether it is stable
> enough to
> > > > > release, and report issues found. My colleagues and I will do
> likewise,
> > > > of
> > > > > course, but no one company can adequately exercise a new release
> with
> > > > this
> > > > > many new contributions.
> > > > >
> > > > > There are two outstanding issue that are not yet committed, but I
> know
> > > > the
> > > > > contributors hope to see in 1.1.0:
> > > > > MAPREDUCE-4049 <
> https://issues.apache.org/jira/browse/MAPREDUCE-4049
> > > > >
> > > > > HADOOP-4012 <https://issues.apache.org/jira/browse/HADOOP-4012
> >
> > > > > Assuming there is an RC-1, and that these two patches can be
> committed
> > > > > during stabilization of RC-0, I will plan to incorporate these
> additional
> > > > > items in RC-1.
> > > > >
> > > > > Best regards,
> > > > > --Matt
> > > > > Release Manager
> > > >
>
>
>
Re: update re: Hadoop-1.1.0-rc0
Posted by Suresh Srinivas <su...@hortonworks.com>.
Matt,
Meanwhile, I am planning to promote the following changes:
HDFS-2465 - Add HDFS support for fadvise readahead and
drop-behind<https://issues.apache.org/jira/browse/HDFS-2465>
HADOOP-Support fadvise and sync_data_range in NativeIO, add ReadaheadPool
class <https://issues.apache.org/jira/browse/HADOOP-7753>
These changes should be low risk - These i/o improvements are turned off by
default in configuration. Brandon Li has done quite a bit of testing
(TestDFSIO, teragen, terasort, teravalidate etc.) on a small cluster (3
nodes) in addition to unit tests.
Regards,
Suresh
On Thu, Jul 12, 2012 at 2:26 PM, Matt Foley <ma...@apache.org> wrote:
> The following additional patches are committed to branch-1.1 and will be in
> 1.1.0:
> - HADOOP-8365 - Add flag to disable durable sync
> - HADOOP-8399 - Remove JDK5 dependency from Hadoop 1.0+ line
> - HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> - HADOOP-8445 - Token should not print the password in toString
> - MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job is
> executed with 0 maps
> - MAPREDUCE-2129 - Job may hang if
> mapreduce.job.committer.setup.cleanup.needed=false and
> mapreduce.map/reduce.failures.maxpercent>0
>
> I've been trying to build and test the RC0, but am having build problems
> with the 32-bit version. Should have it figured out soon.
>
> The following patch is committed to branch-1 and I'll see if I can apply it
> to branch-1.1 before my next build:
> - HDFS-3617 - Port HDFS-96 to branch-1 (support blocks greater than 2GB)
>
> The following patch is committed to branch-1 but had conflicts with
> branch-1.1 and will not be in RC0:
> - HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
>
> The following patches are deferred, as they do not have currently committed
> patches:
> - HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for bzip2)
> - MAPREDUCE-4049 - plugin for generic shuffle service
>
> Thanks,
> --Matt
>
> On Sat, Jul 7, 2012 at 6:32 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Also, I have updated HADOOP-8417 against 1.1.0 and we need to include it.
> > Otherwise, 1.1 will have the same issues for the downstream projects as
> > 1.0.3
> > had.
> >
> > Cos
> >
> > On Sat, Jul 07, 2012 at 05:52PM, Konstantin Boudnik wrote:
> > > Matt,
> > >
> > > Thanks for the update.
> > >
> > > HADOOP-8399 would be beneficial for BigTop release and it is marked for
> > 1.1.0
> > > release. The patch is available for a while now and if someone can
> > review I'd
> > > go ahead and commit it today.
> > >
> > > I am working on the content of 0.3.1 BigTop release and will shortly
> > post the
> > > vote for it. Once Hadoop 1.1 rc is cut we'll start testing it with the
> > rest of
> > > the stack.
> > >
> > > Cos
> > >
> > > On Fri, Jul 06, 2012 at 02:24PM, Matt Foley wrote:
> > > > Hi Cos,
> > > > the query string didn't come thru on the link you sent, but the jira
> > query
> > > > I use is:
> > > > project in (HADOOP,HDFS,MAPREDUCE) and (("Target Version/s" =
> > '1.1.0'
> > > > and (fixVersion != '1.1.0' or fixVersion is EMPTY)) or (fixVersion =
> > > > '1.1.0' and "Target Version/s" is EMPTY)) and (status != Closed and
> > status
> > > > != Resolved) ORDER BY KEY
> > > >
> > > > You're correct that there are quite a few, currently 107, open jiras
> > > > originally targeted for 1.1.0 that do not have committed fixes. Many
> > of
> > > > these are just the inherited backlog of previously identified work.
> I
> > need
> > > > to move them to "Target Version/s" = 1.1.1.
> > > >
> > > > Folks have requested that the following currently open jiras be
> > included in
> > > > 1.1.0:
> > > >
> > > > HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> > > > HADOOP-8445 - Token should not print the password in toString
> > > > HDFS-96 - HDFS does not support blocks greater than 2GB
> > > > HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
> > > > MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job
> > is
> > > > executed with 0 maps
> > > > MAPREDUCE-2129 - Job may hang if
> > > > mapreduce.job.committer.setup.cleanup.needed=false and
> > > > mapreduce.map/reduce.failures.maxpercent>0
> > > > ---
> > > > MAPREDUCE-4049 - plugin for generic shuffle service
> > > > HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for
> > bzip2)
> > > >
> > > > The first six are simple patches that I am comfortable including.
> > > > The last two are complex patches that have not yet been committed.
> > > > I am planning to defer those two to 1.1.1.
> > > >
> > > > Beyond that, I'm going to cut 1.1.0-rc0 from the current state of
> > > > branch-1.1.
> > > > I'm planning to do that this weekend. This is obviously delayed from
> > the
> > > > previous plan, for which I apologize.
> > > >
> > > > Comments welcome.
> > > > --Matt
> > > >
> > > >
> > > > On Tue, Jul 3, 2012 at 8:32 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > >
> > > > > Hi Matt.
> > > > >
> > > > > I am picking up the hat of BigTop's maintainer for Hadoop 1.x line.
> > And I
> > > > > wanted to sync up with about the Hadoop 1.1 release outlook,
> > progress,
> > > > > what help
> > > > > you might need, etc.
> > > > >
> > > > > I see a few jiras left open in the release
> > > > > http://is.gd/OyuaNQ
> > > > > Is this the correct representation of the current status?
> > > > > How I can help from BigTop side (I haven't yet finalized the
> stack's
> > > > > versions), etc. Looking forward for your input. Thanks.
> > > > >
> > > > > Cos
> > > > >
> > > > > On Fri, May 25, 2012 at 02:49PM, Matt Foley wrote:
> > > > > > Greetings. With the approval of a public vote on common-dev@, I
> > have
> > > > > > branched Hadoop branch-1 to create branch-1.1. From this, I will
> > create
> > > > > a
> > > > > > release candidate RC-0 for Hadoop-1.1.0, hopefully to be
> available
> > > > > shortly
> > > > > > after this weekend.
> > > > > >
> > > > > > There are over 80 patches in branch-1, over and above the
> contents
> > of
> > > > > > hadoop-1.0.3. So I anticipate that some stabilization will be
> > needed,
> > > > > > before the RC can be approved as a 1.1.0 release. Your
> > participation in
> > > > > > assuring a stable RC is very important. When it becomes
> available,
> > > > > please
> > > > > > download it and work with it to determine whether it is stable
> > enough to
> > > > > > release, and report issues found. My colleagues and I will do
> > likewise,
> > > > > of
> > > > > > course, but no one company can adequately exercise a new release
> > with
> > > > > this
> > > > > > many new contributions.
> > > > > >
> > > > > > There are two outstanding issue that are not yet committed, but I
> > know
> > > > > the
> > > > > > contributors hope to see in 1.1.0:
> > > > > > MAPREDUCE-4049 <
> > https://issues.apache.org/jira/browse/MAPREDUCE-4049
> > > > > >
> > > > > > HADOOP-4012 <
> https://issues.apache.org/jira/browse/HADOOP-4012
> > >
> > > > > > Assuming there is an RC-1, and that these two patches can be
> > committed
> > > > > > during stabilization of RC-0, I will plan to incorporate these
> > additional
> > > > > > items in RC-1.
> > > > > >
> > > > > > Best regards,
> > > > > > --Matt
> > > > > > Release Manager
> > > > >
> >
> >
> >
>
--
http://hortonworks.com/download/
Re: update re: Hadoop-1.1.0-rc0
Posted by Suresh Srinivas <su...@hortonworks.com>.
Matt,
Meanwhile, I am planning to promote the following changes:
HDFS-2465 - Add HDFS support for fadvise readahead and
drop-behind<https://issues.apache.org/jira/browse/HDFS-2465>
HADOOP-Support fadvise and sync_data_range in NativeIO, add ReadaheadPool
class <https://issues.apache.org/jira/browse/HADOOP-7753>
These changes should be low risk - These i/o improvements are turned off by
default in configuration. Brandon Li has done quite a bit of testing
(TestDFSIO, teragen, terasort, teravalidate etc.) on a small cluster (3
nodes) in addition to unit tests.
Regards,
Suresh
On Thu, Jul 12, 2012 at 2:26 PM, Matt Foley <ma...@apache.org> wrote:
> The following additional patches are committed to branch-1.1 and will be in
> 1.1.0:
> - HADOOP-8365 - Add flag to disable durable sync
> - HADOOP-8399 - Remove JDK5 dependency from Hadoop 1.0+ line
> - HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> - HADOOP-8445 - Token should not print the password in toString
> - MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job is
> executed with 0 maps
> - MAPREDUCE-2129 - Job may hang if
> mapreduce.job.committer.setup.cleanup.needed=false and
> mapreduce.map/reduce.failures.maxpercent>0
>
> I've been trying to build and test the RC0, but am having build problems
> with the 32-bit version. Should have it figured out soon.
>
> The following patch is committed to branch-1 and I'll see if I can apply it
> to branch-1.1 before my next build:
> - HDFS-3617 - Port HDFS-96 to branch-1 (support blocks greater than 2GB)
>
> The following patch is committed to branch-1 but had conflicts with
> branch-1.1 and will not be in RC0:
> - HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
>
> The following patches are deferred, as they do not have currently committed
> patches:
> - HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for bzip2)
> - MAPREDUCE-4049 - plugin for generic shuffle service
>
> Thanks,
> --Matt
>
> On Sat, Jul 7, 2012 at 6:32 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Also, I have updated HADOOP-8417 against 1.1.0 and we need to include it.
> > Otherwise, 1.1 will have the same issues for the downstream projects as
> > 1.0.3
> > had.
> >
> > Cos
> >
> > On Sat, Jul 07, 2012 at 05:52PM, Konstantin Boudnik wrote:
> > > Matt,
> > >
> > > Thanks for the update.
> > >
> > > HADOOP-8399 would be beneficial for BigTop release and it is marked for
> > 1.1.0
> > > release. The patch is available for a while now and if someone can
> > review I'd
> > > go ahead and commit it today.
> > >
> > > I am working on the content of 0.3.1 BigTop release and will shortly
> > post the
> > > vote for it. Once Hadoop 1.1 rc is cut we'll start testing it with the
> > rest of
> > > the stack.
> > >
> > > Cos
> > >
> > > On Fri, Jul 06, 2012 at 02:24PM, Matt Foley wrote:
> > > > Hi Cos,
> > > > the query string didn't come thru on the link you sent, but the jira
> > query
> > > > I use is:
> > > > project in (HADOOP,HDFS,MAPREDUCE) and (("Target Version/s" =
> > '1.1.0'
> > > > and (fixVersion != '1.1.0' or fixVersion is EMPTY)) or (fixVersion =
> > > > '1.1.0' and "Target Version/s" is EMPTY)) and (status != Closed and
> > status
> > > > != Resolved) ORDER BY KEY
> > > >
> > > > You're correct that there are quite a few, currently 107, open jiras
> > > > originally targeted for 1.1.0 that do not have committed fixes. Many
> > of
> > > > these are just the inherited backlog of previously identified work.
> I
> > need
> > > > to move them to "Target Version/s" = 1.1.1.
> > > >
> > > > Folks have requested that the following currently open jiras be
> > included in
> > > > 1.1.0:
> > > >
> > > > HADOOP-8417 - HADOOP-6963 didn't update hadoop-core-pom-template.xml
> > > > HADOOP-8445 - Token should not print the password in toString
> > > > HDFS-96 - HDFS does not support blocks greater than 2GB
> > > > HDFS-3596 - Improve FSEditLog pre-allocation in branch-1
> > > > MAPREDUCE-2903 - Map Tasks graph is throwing XML Parse error when Job
> > is
> > > > executed with 0 maps
> > > > MAPREDUCE-2129 - Job may hang if
> > > > mapreduce.job.committer.setup.cleanup.needed=false and
> > > > mapreduce.map/reduce.failures.maxpercent>0
> > > > ---
> > > > MAPREDUCE-4049 - plugin for generic shuffle service
> > > > HADOOP-7823 - port HADOOP-4012 to branch-1 (splitting support for
> > bzip2)
> > > >
> > > > The first six are simple patches that I am comfortable including.
> > > > The last two are complex patches that have not yet been committed.
> > > > I am planning to defer those two to 1.1.1.
> > > >
> > > > Beyond that, I'm going to cut 1.1.0-rc0 from the current state of
> > > > branch-1.1.
> > > > I'm planning to do that this weekend. This is obviously delayed from
> > the
> > > > previous plan, for which I apologize.
> > > >
> > > > Comments welcome.
> > > > --Matt
> > > >
> > > >
> > > > On Tue, Jul 3, 2012 at 8:32 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > >
> > > > > Hi Matt.
> > > > >
> > > > > I am picking up the hat of BigTop's maintainer for Hadoop 1.x line.
> > And I
> > > > > wanted to sync up with about the Hadoop 1.1 release outlook,
> > progress,
> > > > > what help
> > > > > you might need, etc.
> > > > >
> > > > > I see a few jiras left open in the release
> > > > > http://is.gd/OyuaNQ
> > > > > Is this the correct representation of the current status?
> > > > > How I can help from BigTop side (I haven't yet finalized the
> stack's
> > > > > versions), etc. Looking forward for your input. Thanks.
> > > > >
> > > > > Cos
> > > > >
> > > > > On Fri, May 25, 2012 at 02:49PM, Matt Foley wrote:
> > > > > > Greetings. With the approval of a public vote on common-dev@, I
> > have
> > > > > > branched Hadoop branch-1 to create branch-1.1. From this, I will
> > create
> > > > > a
> > > > > > release candidate RC-0 for Hadoop-1.1.0, hopefully to be
> available
> > > > > shortly
> > > > > > after this weekend.
> > > > > >
> > > > > > There are over 80 patches in branch-1, over and above the
> contents
> > of
> > > > > > hadoop-1.0.3. So I anticipate that some stabilization will be
> > needed,
> > > > > > before the RC can be approved as a 1.1.0 release. Your
> > participation in
> > > > > > assuring a stable RC is very important. When it becomes
> available,
> > > > > please
> > > > > > download it and work with it to determine whether it is stable
> > enough to
> > > > > > release, and report issues found. My colleagues and I will do
> > likewise,
> > > > > of
> > > > > > course, but no one company can adequately exercise a new release
> > with
> > > > > this
> > > > > > many new contributions.
> > > > > >
> > > > > > There are two outstanding issue that are not yet committed, but I
> > know
> > > > > the
> > > > > > contributors hope to see in 1.1.0:
> > > > > > MAPREDUCE-4049 <
> > https://issues.apache.org/jira/browse/MAPREDUCE-4049
> > > > > >
> > > > > > HADOOP-4012 <
> https://issues.apache.org/jira/browse/HADOOP-4012
> > >
> > > > > > Assuming there is an RC-1, and that these two patches can be
> > committed
> > > > > > during stabilization of RC-0, I will plan to incorporate these
> > additional
> > > > > > items in RC-1.
> > > > > >
> > > > > > Best regards,
> > > > > > --Matt
> > > > > > Release Manager
> > > > >
> >
> >
> >
>
--
http://hortonworks.com/download/
Re: update re: Hadoop-1.1.0-rc0
Posted by "Bhandarkar, Milind" <Mi...@emc.com>.
>- MAPREDUCE-4049 - plugin for generic shuffle service
Committers,
What else is needed on MR-4049 to commit ?
- milind
>