You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by Arun C Murthy <ac...@hortonworks.com> on 2014/08/12 22:34:31 UTC

Thinking ahead to hadoop-2.6

Folks,

With hadoop-2.5 nearly done, it's time to start thinking ahead to hadoop-2.6.

Currently, here is the Roadmap per the wiki:

• HADOOP
• Credential provider HADOOP-10607
• HDFS
• Heterogeneous storage (Phase 2) - Support APIs for using storage tiers by the applications HDFS-5682
• Memory as storage tier HDFS-5851
• YARN
• Dynamic Resource Configuration YARN-291
• NodeManager Restart YARN-1336
• ResourceManager HA Phase 2 YARN-556
• Support for admin-specified labels in YARN YARN-796
• Support for automatic, shared cache for YARN application artifacts YARN-1492
• Support NodeGroup layer topology on YARN YARN-18
• Support for Docker containers in YARN YARN-1964
• YARN service registry YARN-913

My suspicion is, as is normal, some will make the cut and some won't. Please do add/subtract from the list as appropriate. Ideally, it would be good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence.

More importantly, as we discussed previously, we'd like hadoop-2.6 to be the *last* Apache Hadoop 2.x release which support JDK6. I'll start a discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see how they feel about this.

thanks,
Arun

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

I think we should delay cutting the branch beyond the end of the week then,
seeing as the HDFS-6581 merge vote (assuming it goes through) doesn't close
until 12am on Tuesday next week. Archival storage is also still finishing
up user APIs and I had a few other q's posted on HDFS-7081, so again might
not make it by the end of the week.

I don't think it'd be too bad to retarget these to 2.7 though, since Arun
said he plans to roll 2.7 shortly after 2.6. I think a few more weeks would
really seal the deal.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> I actually would like to see both archival storage and single replica
> memory writes to be in 2.6 release. Archival storage is in the final stages
> of getting ready for branch-2 merge as Nicholas has already indicated on
> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> features are being in development for sometime.
>
> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Arun,
> >
> > Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> > things accordingly?
> >
> > I think the KMS and transparent encryption are ready to go. We've got a
> > very few further bug fixes pending, but that's it.
> >
> > Two HDFS things that I think probably won't make the end of the week are
> > archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > trunk and I think needs a little more work before it goes into branch-2.
> > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> > yet.
> >
> > Just my 2c as I did not work directly on these features. I just generally
> > shy away from shipping bits quite this fresh.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >
> > > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> > >
> > > I'll create branch-2.6 by end of the week and we can go from there to
> > > stabilize it - hopefully in the next few weeks.
> > >
> > > Thoughts?
> > >
> > > thanks,
> > > Arun
> > >
> > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > hadoop-2.6.
> > > >
> > > >  Currently, here is the Roadmap per the wiki:
> > > >
> > > >         • HADOOP
> > > >                 • Credential provider HADOOP-10607
> > > >         • HDFS
> > > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > > using
> > > > storage tiers by the applications HDFS-5682
> > > >                 • Memory as storage tier HDFS-5851
> > > >         • YARN
> > > >                 • Dynamic Resource Configuration YARN-291
> > > >                 • NodeManager Restart YARN-1336
> > > >                 • ResourceManager HA Phase 2 YARN-556
> > > >                 • Support for admin-specified labels in YARN YARN-796
> > > >                 • Support for automatic, shared cache for YARN
> > > application
> > > > artifacts YARN-1492
> > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > >                 • Support for Docker containers in YARN YARN-1964
> > > >                 • YARN service registry YARN-913
> > > >
> > > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> > > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> > be
> > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > cadence.
> > > >
> > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> > be
> > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> > see
> > > > how they feel about this.
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Jing Zhao <ji...@hortonworks.com>.

Just to give a quick update about the current status of HDFS-6584 (archival
storage). So far after HDFS-7081 got committed this morning, the main
functionalities have already been finished. As a summary two new
DistributedFileSystem APIs are added: getStoragePolicies and
setStoragePolicy. We have also been doing system tests for weeks and we
will continue testing. There are still one or two pending issues but we're
actively working on it. So I'm pretty confident that the archival storage
work can get ready given the current plan to release 2.6 next week.

On Wed, Sep 24, 2014 at 3:35 PM, Jitendra Pandey <ji...@hortonworks.com>
wrote:

> I also believe its worth a week's wait to include HDFS-6584 and HDFS-6581
> in 2.6.
>
> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
> > Given some of the features are in final stages of stabilization,
> > Arun, we should hold off creating 2.6 branch or building an RC by a week?
> > All the features in flux are important ones and worth delaying the
> release
> > by a week.
> >
> > On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > Hey Nicholas,
> > >
> > > My concern about Archival Storage isn't related to the code quality or
> > the
> > > size of the feature. I think that you and Jing did good work. My
> concern
> > is
> > > that once we ship, we're locked into that set of archival storage APIs,
> > and
> > > these APIs are not yet finalized. Simply being able to turn off the
> > feature
> > > does not change the compatibility story.
> > >
> > > I'm willing to devote time to help review these JIRAs and kick the
> tires
> > on
> > > the APIs, but my point above was that I'm not sure it'd all be done by
> > the
> > > end of the week. Testing might also reveal additional changes that need
> > to
> > > be made, which also might not happen by end-of-week.
> > >
> > > I guess the question before us is if we're comfortable putting
> something
> > in
> > > branch-2.6 and then potentially adding API changes after. I'm okay with
> > > that as long as we're all aware that this might happen.
> > >
> > > Arun, as RM is this cool with you? Again, I like this feature and I'm
> > fine
> > > with it's inclusion, just a heads up that we might need some extra time
> > to
> > > finalize things before an RC can be cut.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> > > s29752-hadoopdev@yahoo.com.invalid> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am worry about KMS and transparent encryption since there are quite
> > > many
> > > > bugs discovered after it got merged to branch-2.  It gives us an
> > > impression
> > > > that the feature is not yet well tested.  Indeed, transparent
> > encryption
> > > is
> > > > a complicated feature which changes the core part of HDFS.  It is not
> > > easy
> > > > to get everything right.
> > > >
> > > >
> > > > For HDFS-6584: Archival Storage, it is a relatively simple and low
> risk
> > > > feature.  It introduces a new storage type ARCHIVE and the concept of
> > > block
> > > > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> > > storage,
> > > > the blocks will be stored using the appropriate storage types
> specified
> > > by
> > > > storage policies assigned to the files/directories.  Cluster admin
> > could
> > > > disable the feature by simply not configuring any storage type and
> not
> > > > setting any storage policy as before.   As Suresh mentioned,
> HDFS-6584
> > is
> > > > in the final stages to be merged to branch-2.
> > > >
> > > > Regards,
> > > > Tsz-Wo
> > > >
> > > >
> > > >
> > > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > > > suresh@hortonworks.com> wrote:
> > > >
> > > >
> > > > >
> > > > >
> > > > >I actually would like to see both archival storage and single
> replica
> > > > >memory writes to be in 2.6 release. Archival storage is in the final
> > > > stages
> > > > >of getting ready for branch-2 merge as Nicholas has already
> indicated
> > on
> > > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > > these
> > > > >features are being in development for sometime.
> > > > >
> > > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > > > >wrote:
> > > > >
> > > > >> Hey Arun,
> > > > >>
> > > > >> Maybe we could do a quick run through of the Roadmap wiki and
> > > > add/retarget
> > > > >> things accordingly?
> > > > >>
> > > > >> I think the KMS and transparent encryption are ready to go. We've
> > got
> > > a
> > > > >> very few further bug fixes pending, but that's it.
> > > > >>
> > > > >> Two HDFS things that I think probably won't make the end of the
> week
> > > are
> > > > >> archival storage (HDFS-6584) and single replica memory writes
> > > > (HDFS-6581),
> > > > >> which I believe are under the HSM banner. HDFS-6484 was just
> merged
> > to
> > > > >> trunk and I think needs a little more work before it goes into
> > > branch-2.
> > > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> > further
> > > > off
> > > > >> yet.
> > > > >>
> > > > >> Just my 2c as I did not work directly on these features. I just
> > > > generally
> > > > >> shy away from shipping bits quite this fresh.
> > > > >>
> > > > >> Thanks,
> > > > >> Andrew
> > > > >>
> > > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <acm@hortonworks.com
> >
> > > > wrote:
> > > > >>
> > > > >> > Looks like most of the content is in and hadoop-2.6 is shaping
> up
> > > > nicely.
> > > > >> >
> > > > >> > I'll create branch-2.6 by end of the week and we can go from
> there
> > > to
> > > > >> > stabilize it - hopefully in the next few weeks.
> > > > >> >
> > > > >> > Thoughts?
> > > > >> >
> > > > >> > thanks,
> > > > >> > Arun
> > > > >> >
> > > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> > acm@hortonworks.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Folks,
> > > > >> > >
> > > > >> > >  With hadoop-2.5 nearly done, it's time to start thinking
> ahead
> > to
> > > > >> > > hadoop-2.6.
> > > > >> > >
> > > > >> > >  Currently, here is the Roadmap per the wiki:
> > > > >> > >
> > > > >> > >         • HADOOP
> > > > >> > >                 • Credential provider HADOOP-10607
> > > > >> > >         • HDFS
> > > > >> > >                 • Heterogeneous storage (Phase 2) - Support
> APIs
> > > for
> > > > >> > using
> > > > >> > > storage tiers by the applications HDFS-5682
> > > > >> > >                 • Memory as storage tier HDFS-5851
> > > > >> > >         • YARN
> > > > >> > >                 • Dynamic Resource Configuration YARN-291
> > > > >> > >                 • NodeManager Restart YARN-1336
> > > > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > > > >> > >                 • Support for admin-specified labels in YARN
> > > > YARN-796
> > > > >> > >                 • Support for automatic, shared cache for YARN
> > > > >> > application
> > > > >> > > artifacts YARN-1492
> > > > >> > >                 • Support NodeGroup layer topology on YARN
> > YARN-18
> > > > >> > >                 • Support for Docker containers in YARN
> > YARN-1964
> > > > >> > >                 • YARN service registry YARN-913
> > > > >> > >
> > > > >> > >  My suspicion is, as is normal, some will make the cut and
> some
> > > > won't.
> > > > >> > > Please do add/subtract from the list as appropriate. Ideally,
> it
> > > > would
> > > > >> be
> > > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> > up a
> > > > >> > cadence.
> > > > >> > >
> > > > >> > >  More importantly, as we discussed previously, we'd like
> > > hadoop-2.6
> > > > to
> > > > >> be
> > > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > > start
> > > > a
> > > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie
> etc.)
> > > and
> > > > >> see
> > > > >> > > how they feel about this.
> > > > >> > >
> > > > >> > > thanks,
> > > > >> > > Arun
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> >
> > > > >> > --
> > > > >> > Arun C. Murthy
> > > > >> > Hortonworks Inc.
> > > > >> > http://hortonworks.com/
> > > > >> >
> > > > >> > --
> > > > >> > CONFIDENTIALITY NOTICE
> > > > >> > NOTICE: This message is intended for the use of the individual
> or
> > > > entity
> > > > >> to
> > > > >> > which it is addressed and may contain information that is
> > > > confidential,
> > > > >> > privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > >> > of this message is not the intended recipient, you are hereby
> > > notified
> > > > >> that
> > > > >> > any printing, copying, dissemination, distribution, disclosure
> or
> > > > >> > forwarding of this communication is strictly prohibited. If you
> > have
> > > > >> > received this communication in error, please contact the sender
> > > > >> immediately
> > > > >> > and delete it from your system. Thank You.
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > >--
> > > > >http://hortonworks.com/download/
> > > > >
> > > > >--
> > > > >CONFIDENTIALITY NOTICE
> > > > >NOTICE: This message is intended for the use of the individual or
> > entity
> > > > to
> > > > >which it is addressed and may contain information that is
> > confidential,
> > > > >privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > >of this message is not the intended recipient, you are hereby
> notified
> > > > that
> > > > >any printing, copying, dissemination, distribution, disclosure or
> > > > >forwarding of this communication is strictly prohibited. If you have
> > > > >received this communication in error, please contact the sender
> > > > immediately
> > > > >and delete it from your system. Thank You.
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> <http://hortonworks.com/download/>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Jing Zhao <ji...@hortonworks.com>.

Just to give a quick update about the current status of HDFS-6584 (archival
storage). So far after HDFS-7081 got committed this morning, the main
functionalities have already been finished. As a summary two new
DistributedFileSystem APIs are added: getStoragePolicies and
setStoragePolicy. We have also been doing system tests for weeks and we
will continue testing. There are still one or two pending issues but we're
actively working on it. So I'm pretty confident that the archival storage
work can get ready given the current plan to release 2.6 next week.

On Wed, Sep 24, 2014 at 3:35 PM, Jitendra Pandey <ji...@hortonworks.com>
wrote:

> I also believe its worth a week's wait to include HDFS-6584 and HDFS-6581
> in 2.6.
>
> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
> > Given some of the features are in final stages of stabilization,
> > Arun, we should hold off creating 2.6 branch or building an RC by a week?
> > All the features in flux are important ones and worth delaying the
> release
> > by a week.
> >
> > On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > Hey Nicholas,
> > >
> > > My concern about Archival Storage isn't related to the code quality or
> > the
> > > size of the feature. I think that you and Jing did good work. My
> concern
> > is
> > > that once we ship, we're locked into that set of archival storage APIs,
> > and
> > > these APIs are not yet finalized. Simply being able to turn off the
> > feature
> > > does not change the compatibility story.
> > >
> > > I'm willing to devote time to help review these JIRAs and kick the
> tires
> > on
> > > the APIs, but my point above was that I'm not sure it'd all be done by
> > the
> > > end of the week. Testing might also reveal additional changes that need
> > to
> > > be made, which also might not happen by end-of-week.
> > >
> > > I guess the question before us is if we're comfortable putting
> something
> > in
> > > branch-2.6 and then potentially adding API changes after. I'm okay with
> > > that as long as we're all aware that this might happen.
> > >
> > > Arun, as RM is this cool with you? Again, I like this feature and I'm
> > fine
> > > with it's inclusion, just a heads up that we might need some extra time
> > to
> > > finalize things before an RC can be cut.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> > > s29752-hadoopdev@yahoo.com.invalid> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am worry about KMS and transparent encryption since there are quite
> > > many
> > > > bugs discovered after it got merged to branch-2.  It gives us an
> > > impression
> > > > that the feature is not yet well tested.  Indeed, transparent
> > encryption
> > > is
> > > > a complicated feature which changes the core part of HDFS.  It is not
> > > easy
> > > > to get everything right.
> > > >
> > > >
> > > > For HDFS-6584: Archival Storage, it is a relatively simple and low
> risk
> > > > feature.  It introduces a new storage type ARCHIVE and the concept of
> > > block
> > > > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> > > storage,
> > > > the blocks will be stored using the appropriate storage types
> specified
> > > by
> > > > storage policies assigned to the files/directories.  Cluster admin
> > could
> > > > disable the feature by simply not configuring any storage type and
> not
> > > > setting any storage policy as before.   As Suresh mentioned,
> HDFS-6584
> > is
> > > > in the final stages to be merged to branch-2.
> > > >
> > > > Regards,
> > > > Tsz-Wo
> > > >
> > > >
> > > >
> > > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > > > suresh@hortonworks.com> wrote:
> > > >
> > > >
> > > > >
> > > > >
> > > > >I actually would like to see both archival storage and single
> replica
> > > > >memory writes to be in 2.6 release. Archival storage is in the final
> > > > stages
> > > > >of getting ready for branch-2 merge as Nicholas has already
> indicated
> > on
> > > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > > these
> > > > >features are being in development for sometime.
> > > > >
> > > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > > > >wrote:
> > > > >
> > > > >> Hey Arun,
> > > > >>
> > > > >> Maybe we could do a quick run through of the Roadmap wiki and
> > > > add/retarget
> > > > >> things accordingly?
> > > > >>
> > > > >> I think the KMS and transparent encryption are ready to go. We've
> > got
> > > a
> > > > >> very few further bug fixes pending, but that's it.
> > > > >>
> > > > >> Two HDFS things that I think probably won't make the end of the
> week
> > > are
> > > > >> archival storage (HDFS-6584) and single replica memory writes
> > > > (HDFS-6581),
> > > > >> which I believe are under the HSM banner. HDFS-6484 was just
> merged
> > to
> > > > >> trunk and I think needs a little more work before it goes into
> > > branch-2.
> > > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> > further
> > > > off
> > > > >> yet.
> > > > >>
> > > > >> Just my 2c as I did not work directly on these features. I just
> > > > generally
> > > > >> shy away from shipping bits quite this fresh.
> > > > >>
> > > > >> Thanks,
> > > > >> Andrew
> > > > >>
> > > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <acm@hortonworks.com
> >
> > > > wrote:
> > > > >>
> > > > >> > Looks like most of the content is in and hadoop-2.6 is shaping
> up
> > > > nicely.
> > > > >> >
> > > > >> > I'll create branch-2.6 by end of the week and we can go from
> there
> > > to
> > > > >> > stabilize it - hopefully in the next few weeks.
> > > > >> >
> > > > >> > Thoughts?
> > > > >> >
> > > > >> > thanks,
> > > > >> > Arun
> > > > >> >
> > > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> > acm@hortonworks.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Folks,
> > > > >> > >
> > > > >> > >  With hadoop-2.5 nearly done, it's time to start thinking
> ahead
> > to
> > > > >> > > hadoop-2.6.
> > > > >> > >
> > > > >> > >  Currently, here is the Roadmap per the wiki:
> > > > >> > >
> > > > >> > >         • HADOOP
> > > > >> > >                 • Credential provider HADOOP-10607
> > > > >> > >         • HDFS
> > > > >> > >                 • Heterogeneous storage (Phase 2) - Support
> APIs
> > > for
> > > > >> > using
> > > > >> > > storage tiers by the applications HDFS-5682
> > > > >> > >                 • Memory as storage tier HDFS-5851
> > > > >> > >         • YARN
> > > > >> > >                 • Dynamic Resource Configuration YARN-291
> > > > >> > >                 • NodeManager Restart YARN-1336
> > > > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > > > >> > >                 • Support for admin-specified labels in YARN
> > > > YARN-796
> > > > >> > >                 • Support for automatic, shared cache for YARN
> > > > >> > application
> > > > >> > > artifacts YARN-1492
> > > > >> > >                 • Support NodeGroup layer topology on YARN
> > YARN-18
> > > > >> > >                 • Support for Docker containers in YARN
> > YARN-1964
> > > > >> > >                 • YARN service registry YARN-913
> > > > >> > >
> > > > >> > >  My suspicion is, as is normal, some will make the cut and
> some
> > > > won't.
> > > > >> > > Please do add/subtract from the list as appropriate. Ideally,
> it
> > > > would
> > > > >> be
> > > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> > up a
> > > > >> > cadence.
> > > > >> > >
> > > > >> > >  More importantly, as we discussed previously, we'd like
> > > hadoop-2.6
> > > > to
> > > > >> be
> > > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > > start
> > > > a
> > > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie
> etc.)
> > > and
> > > > >> see
> > > > >> > > how they feel about this.
> > > > >> > >
> > > > >> > > thanks,
> > > > >> > > Arun
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> >
> > > > >> > --
> > > > >> > Arun C. Murthy
> > > > >> > Hortonworks Inc.
> > > > >> > http://hortonworks.com/
> > > > >> >
> > > > >> > --
> > > > >> > CONFIDENTIALITY NOTICE
> > > > >> > NOTICE: This message is intended for the use of the individual
> or
> > > > entity
> > > > >> to
> > > > >> > which it is addressed and may contain information that is
> > > > confidential,
> > > > >> > privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > >> > of this message is not the intended recipient, you are hereby
> > > notified
> > > > >> that
> > > > >> > any printing, copying, dissemination, distribution, disclosure
> or
> > > > >> > forwarding of this communication is strictly prohibited. If you
> > have
> > > > >> > received this communication in error, please contact the sender
> > > > >> immediately
> > > > >> > and delete it from your system. Thank You.
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > >--
> > > > >http://hortonworks.com/download/
> > > > >
> > > > >--
> > > > >CONFIDENTIALITY NOTICE
> > > > >NOTICE: This message is intended for the use of the individual or
> > entity
> > > > to
> > > > >which it is addressed and may contain information that is
> > confidential,
> > > > >privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > >of this message is not the intended recipient, you are hereby
> notified
> > > > that
> > > > >any printing, copying, dissemination, distribution, disclosure or
> > > > >forwarding of this communication is strictly prohibited. If you have
> > > > >received this communication in error, please contact the sender
> > > > immediately
> > > > >and delete it from your system. Thank You.
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> <http://hortonworks.com/download/>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Jitendra Pandey <ji...@hortonworks.com>.

I also believe its worth a week's wait to include HDFS-6584 and HDFS-6581
in 2.6.

On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> Given some of the features are in final stages of stabilization,
> Arun, we should hold off creating 2.6 branch or building an RC by a week?
> All the features in flux are important ones and worth delaying the release
> by a week.
>
> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Nicholas,
> >
> > My concern about Archival Storage isn't related to the code quality or
> the
> > size of the feature. I think that you and Jing did good work. My concern
> is
> > that once we ship, we're locked into that set of archival storage APIs,
> and
> > these APIs are not yet finalized. Simply being able to turn off the
> feature
> > does not change the compatibility story.
> >
> > I'm willing to devote time to help review these JIRAs and kick the tires
> on
> > the APIs, but my point above was that I'm not sure it'd all be done by
> the
> > end of the week. Testing might also reveal additional changes that need
> to
> > be made, which also might not happen by end-of-week.
> >
> > I guess the question before us is if we're comfortable putting something
> in
> > branch-2.6 and then potentially adding API changes after. I'm okay with
> > that as long as we're all aware that this might happen.
> >
> > Arun, as RM is this cool with you? Again, I like this feature and I'm
> fine
> > with it's inclusion, just a heads up that we might need some extra time
> to
> > finalize things before an RC can be cut.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> > s29752-hadoopdev@yahoo.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > I am worry about KMS and transparent encryption since there are quite
> > many
> > > bugs discovered after it got merged to branch-2.  It gives us an
> > impression
> > > that the feature is not yet well tested.  Indeed, transparent
> encryption
> > is
> > > a complicated feature which changes the core part of HDFS.  It is not
> > easy
> > > to get everything right.
> > >
> > >
> > > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > > feature.  It introduces a new storage type ARCHIVE and the concept of
> > block
> > > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> > storage,
> > > the blocks will be stored using the appropriate storage types specified
> > by
> > > storage policies assigned to the files/directories.  Cluster admin
> could
> > > disable the feature by simply not configuring any storage type and not
> > > setting any storage policy as before.   As Suresh mentioned, HDFS-6584
> is
> > > in the final stages to be merged to branch-2.
> > >
> > > Regards,
> > > Tsz-Wo
> > >
> > >
> > >
> > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > > suresh@hortonworks.com> wrote:
> > >
> > >
> > > >
> > > >
> > > >I actually would like to see both archival storage and single replica
> > > >memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > >of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > >features are being in development for sometime.
> > > >
> > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > >wrote:
> > > >
> > > >> Hey Arun,
> > > >>
> > > >> Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > >> things accordingly?
> > > >>
> > > >> I think the KMS and transparent encryption are ready to go. We've
> got
> > a
> > > >> very few further bug fixes pending, but that's it.
> > > >>
> > > >> Two HDFS things that I think probably won't make the end of the week
> > are
> > > >> archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > >> which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > >> trunk and I think needs a little more work before it goes into
> > branch-2.
> > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > >> yet.
> > > >>
> > > >> Just my 2c as I did not work directly on these features. I just
> > > generally
> > > >> shy away from shipping bits quite this fresh.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >>
> > > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > >> >
> > > >> > I'll create branch-2.6 by end of the week and we can go from there
> > to
> > > >> > stabilize it - hopefully in the next few weeks.
> > > >> >
> > > >> > Thoughts?
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Folks,
> > > >> > >
> > > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > >> > > hadoop-2.6.
> > > >> > >
> > > >> > >  Currently, here is the Roadmap per the wiki:
> > > >> > >
> > > >> > >         • HADOOP
> > > >> > >                 • Credential provider HADOOP-10607
> > > >> > >         • HDFS
> > > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > >> > using
> > > >> > > storage tiers by the applications HDFS-5682
> > > >> > >                 • Memory as storage tier HDFS-5851
> > > >> > >         • YARN
> > > >> > >                 • Dynamic Resource Configuration YARN-291
> > > >> > >                 • NodeManager Restart YARN-1336
> > > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > > >> > >                 • Support for admin-specified labels in YARN
> > > YARN-796
> > > >> > >                 • Support for automatic, shared cache for YARN
> > > >> > application
> > > >> > > artifacts YARN-1492
> > > >> > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > >> > >                 • Support for Docker containers in YARN
> YARN-1964
> > > >> > >                 • YARN service registry YARN-913
> > > >> > >
> > > >> > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > >> be
> > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up a
> > > >> > cadence.
> > > >> > >
> > > >> > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > to
> > > >> be
> > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start
> > > a
> > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > >> see
> > > >> > > how they feel about this.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > >> to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > >--
> > > >http://hortonworks.com/download/
> > > >
> > > >--
> > > >CONFIDENTIALITY NOTICE
> > > >NOTICE: This message is intended for the use of the individual or
> entity
> > > to
> > > >which it is addressed and may contain information that is
> confidential,
> > > >privileged and exempt from disclosure under applicable law. If the
> > reader
> > > >of this message is not the intended recipient, you are hereby notified
> > > that
> > > >any printing, copying, dissemination, distribution, disclosure or
> > > >forwarding of this communication is strictly prohibited. If you have
> > > >received this communication in error, please contact the sender
> > > immediately
> > > >and delete it from your system. Thank You.
> > > >
> > > >
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
<http://hortonworks.com/download/>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

Re-sending… doesn't look like it made through.

On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
> 
>  I've created branch-2.6 to stabilize the release.
>  
>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
> 
>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
> 
> thanks,
> Arun
> 
> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> 
>> Sounds good. I'll branch this weekend and we can merge the jiras we
>> discussed in this thread as they they get wrapped next week.
>> 
>> Thanks everyone.
>> 
>> Arun
>> 
>> 
>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> We can branch off in a week or two so that work on branch-2 itself can go
>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>> can then decide on the timeline of the release candidates once branch-2.6
>>> is close to being done w.r.t the planned features.
>>> 
>>> Branching it off can let us focus on specific features that we want in for
>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>> trivial pain of committing to one more branch, but it's worth it in this
>>> case IMO.
>>> 
>>> A lot of efforts are happening in parallel from the YARN side from where I
>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>> a perpetual delay.
>>> 
>>> My 2 cents.
>>> 
>>> +Vinod
>>> 
>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>> wrote:
>>> 
>>>> Given some of the features are in final stages of stabilization,
>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>> All the features in flux are important ones and worth delaying the release
>>>> by a week.
>>>> 
>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> Hey Nicholas,
>>>>> 
>>>>> My concern about Archival Storage isn't related to the code quality or
>>>> the
>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>> is
>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>> and
>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>> feature
>>>>> does not change the compatibility story.
>>>>> 
>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>> on
>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>> the
>>>>> end of the week. Testing might also reveal additional changes that need
>>>> to
>>>>> be made, which also might not happen by end-of-week.
>>>>> 
>>>>> I guess the question before us is if we're comfortable putting something
>>>> in
>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>> that as long as we're all aware that this might happen.
>>>>> 
>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>> fine
>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>> to
>>>>> finalize things before an RC can be cut.
>>>>> 
>>>>> Thanks,
>>>>> Andrew
>>>>> 
>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>> many
>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>> impression
>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>> encryption
>>>>> is
>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>> easy
>>>>>> to get everything right.
>>>>>> 
>>>>>> 
>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>> block
>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>> storage,
>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>> by
>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>> could
>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>> is
>>>>>> in the final stages to be merged to branch-2.
>>>>>> 
>>>>>> Regards,
>>>>>> Tsz-Wo
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>> suresh@hortonworks.com> wrote:
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>> stages
>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>> on
>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>> these
>>>>>>> features are being in development for sometime.
>>>>>>> 
>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>> andrew.wang@cloudera.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey Arun,
>>>>>>>> 
>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>> add/retarget
>>>>>>>> things accordingly?
>>>>>>>> 
>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>> got
>>>>> a
>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>> 
>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>> are
>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>> (HDFS-6581),
>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>> to
>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>> branch-2.
>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>> further
>>>>>> off
>>>>>>>> yet.
>>>>>>>> 
>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>> generally
>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>> nicely.
>>>>>>>>> 
>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>> to
>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>> acm@hortonworks.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Folks,
>>>>>>>>>> 
>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>> to
>>>>>>>>>> hadoop-2.6.
>>>>>>>>>> 
>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>> 
>>>>>>>>>>       • HADOOP
>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>       • HDFS
>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>> for
>>>>>>>>> using
>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>       • YARN
>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>> YARN-796
>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>> application
>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>> YARN-18
>>>>>>>>>>               • Support for Docker containers in YARN
>>>> YARN-1964
>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>> 
>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>> won't.
>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>> would
>>>>>>>> be
>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>> up a
>>>>>>>>> cadence.
>>>>>>>>>> 
>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>> hadoop-2.6
>>>>>> to
>>>>>>>> be
>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>> start
>>>>>> a
>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>> and
>>>>>>>> see
>>>>>>>>>> how they feel about this.
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity
>>>>>>>> to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>> 
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> http://hortonworks.com/download/
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Neeraj Asrani <ne...@gmail.com>.

Hi Arun,

Can you assign one of the issues to me. Il be glad to help out.

Thanks,
Nick.


On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> 2.6.0 is close now.
>
> Here are the remaining blockers, I'm hoping cut an RC in the next week or
> so:
> http://s.apache.org/hadoop-2.6.0-blockers
>
> thanks,
> Arun
>
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > Folks,
> >
> >  I've created branch-2.6 to stabilize the release.
> >
> >  Committers, please exercise caution henceforth on commits other than
> the ones we've discussed on this thread already.
> >
> >  By default new features should now be targeted to the version "2.7"
> henceforth - I've ensure all the projects have that version on jira.
> >
> > thanks,
> > Arun
> >
> > On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> >
> >> Sounds good. I'll branch this weekend and we can merge the jiras we
> >> discussed in this thread as they they get wrapped next week.
> >>
> >> Thanks everyone.
> >>
> >> Arun
> >>
> >>
> >>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >>>
> >>> We can branch off in a week or two so that work on branch-2 itself can
> go
> >>> ahead with other features that can't fit in 2.6. Independent of that,
> we
> >>> can then decide on the timeline of the release candidates once
> branch-2.6
> >>> is close to being done w.r.t the planned features.
> >>>
> >>> Branching it off can let us focus on specific features that we want in
> for
> >>> 2.6 and then eventually blockers for the release, nothing else. There
> is a
> >>> trivial pain of committing to one more branch, but it's worth it in
> this
> >>> case IMO.
> >>>
> >>> A lot of efforts are happening in parallel from the YARN side from
> where I
> >>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if
> we
> >>> don't branch off and selectively try to get stuff in, it is likely to
> be in
> >>> a perpetual delay.
> >>>
> >>> My 2 cents.
> >>>
> >>> +Vinod
> >>>
> >>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <
> suresh@hortonworks.com>
> >>> wrote:
> >>>
> >>>> Given some of the features are in final stages of stabilization,
> >>>> Arun, we should hold off creating 2.6 branch or building an RC by a
> week?
> >>>> All the features in flux are important ones and worth delaying the
> release
> >>>> by a week.
> >>>>
> >>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> Hey Nicholas,
> >>>>>
> >>>>> My concern about Archival Storage isn't related to the code quality
> or
> >>>> the
> >>>>> size of the feature. I think that you and Jing did good work. My
> concern
> >>>> is
> >>>>> that once we ship, we're locked into that set of archival storage
> APIs,
> >>>> and
> >>>>> these APIs are not yet finalized. Simply being able to turn off the
> >>>> feature
> >>>>> does not change the compatibility story.
> >>>>>
> >>>>> I'm willing to devote time to help review these JIRAs and kick the
> tires
> >>>> on
> >>>>> the APIs, but my point above was that I'm not sure it'd all be done
> by
> >>>> the
> >>>>> end of the week. Testing might also reveal additional changes that
> need
> >>>> to
> >>>>> be made, which also might not happen by end-of-week.
> >>>>>
> >>>>> I guess the question before us is if we're comfortable putting
> something
> >>>> in
> >>>>> branch-2.6 and then potentially adding API changes after. I'm okay
> with
> >>>>> that as long as we're all aware that this might happen.
> >>>>>
> >>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
> >>>> fine
> >>>>> with it's inclusion, just a heads up that we might need some extra
> time
> >>>> to
> >>>>> finalize things before an RC can be cut.
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>>
> >>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> >>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am worry about KMS and transparent encryption since there are
> quite
> >>>>> many
> >>>>>> bugs discovered after it got merged to branch-2.  It gives us an
> >>>>> impression
> >>>>>> that the feature is not yet well tested.  Indeed, transparent
> >>>> encryption
> >>>>> is
> >>>>>> a complicated feature which changes the core part of HDFS.  It is
> not
> >>>>> easy
> >>>>>> to get everything right.
> >>>>>>
> >>>>>>
> >>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low
> risk
> >>>>>> feature.  It introduces a new storage type ARCHIVE and the concept
> of
> >>>>> block
> >>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
> >>>>> storage,
> >>>>>> the blocks will be stored using the appropriate storage types
> specified
> >>>>> by
> >>>>>> storage policies assigned to the files/directories.  Cluster admin
> >>>> could
> >>>>>> disable the feature by simply not configuring any storage type and
> not
> >>>>>> setting any storage policy as before.   As Suresh mentioned,
> HDFS-6584
> >>>> is
> >>>>>> in the final stages to be merged to branch-2.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Tsz-Wo
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> >>>>>> suresh@hortonworks.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I actually would like to see both archival storage and single
> replica
> >>>>>>> memory writes to be in 2.6 release. Archival storage is in the
> final
> >>>>>> stages
> >>>>>>> of getting ready for branch-2 merge as Nicholas has already
> indicated
> >>>> on
> >>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both
> of
> >>>>> these
> >>>>>>> features are being in development for sometime.
> >>>>>>>
> >>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> >>>> andrew.wang@cloudera.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hey Arun,
> >>>>>>>>
> >>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
> >>>>>> add/retarget
> >>>>>>>> things accordingly?
> >>>>>>>>
> >>>>>>>> I think the KMS and transparent encryption are ready to go. We've
> >>>> got
> >>>>> a
> >>>>>>>> very few further bug fixes pending, but that's it.
> >>>>>>>>
> >>>>>>>> Two HDFS things that I think probably won't make the end of the
> week
> >>>>> are
> >>>>>>>> archival storage (HDFS-6584) and single replica memory writes
> >>>>>> (HDFS-6581),
> >>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just
> merged
> >>>> to
> >>>>>>>> trunk and I think needs a little more work before it goes into
> >>>>> branch-2.
> >>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> >>>> further
> >>>>>> off
> >>>>>>>> yet.
> >>>>>>>>
> >>>>>>>> Just my 2c as I did not work directly on these features. I just
> >>>>>> generally
> >>>>>>>> shy away from shipping bits quite this fresh.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <acm@hortonworks.com
> >
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
> >>>>>> nicely.
> >>>>>>>>>
> >>>>>>>>> I'll create branch-2.6 by end of the week and we can go from
> there
> >>>>> to
> >>>>>>>>> stabilize it - hopefully in the next few weeks.
> >>>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> >>>> acm@hortonworks.com
> >>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Folks,
> >>>>>>>>>>
> >>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
> >>>> to
> >>>>>>>>>> hadoop-2.6.
> >>>>>>>>>>
> >>>>>>>>>> Currently, here is the Roadmap per the wiki:
> >>>>>>>>>>
> >>>>>>>>>>       • HADOOP
> >>>>>>>>>>               • Credential provider HADOOP-10607
> >>>>>>>>>>       • HDFS
> >>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
> >>>>> for
> >>>>>>>>> using
> >>>>>>>>>> storage tiers by the applications HDFS-5682
> >>>>>>>>>>               • Memory as storage tier HDFS-5851
> >>>>>>>>>>       • YARN
> >>>>>>>>>>               • Dynamic Resource Configuration YARN-291
> >>>>>>>>>>               • NodeManager Restart YARN-1336
> >>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
> >>>>>>>>>>               • Support for admin-specified labels in YARN
> >>>>>> YARN-796
> >>>>>>>>>>               • Support for automatic, shared cache for YARN
> >>>>>>>>> application
> >>>>>>>>>> artifacts YARN-1492
> >>>>>>>>>>               • Support NodeGroup layer topology on YARN
> >>>> YARN-18
> >>>>>>>>>>               • Support for Docker containers in YARN
> >>>> YARN-1964
> >>>>>>>>>>               • YARN service registry YARN-913
> >>>>>>>>>>
> >>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
> >>>>>> won't.
> >>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
> >>>>>> would
> >>>>>>>> be
> >>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> >>>> up a
> >>>>>>>>> cadence.
> >>>>>>>>>>
> >>>>>>>>>> More importantly, as we discussed previously, we'd like
> >>>>> hadoop-2.6
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
> >>>>> start
> >>>>>> a
> >>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> >>>>> and
> >>>>>>>> see
> >>>>>>>>>> how they feel about this.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>> entity
> >>>>>>>> to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> the
> >>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> http://hortonworks.com/download/
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>>> to
> >>>>>>> which it is addressed and may contain information that is
> >>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://hortonworks.com/download/
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>>> of this message is not the intended recipient, you are hereby
> notified that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> immediately
> >>>> and delete it from your system. Thank You.
> >>>
> >>> --
> >>> CONFIDENTIALITY NOTICE
> >>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>> which it is addressed and may contain information that is confidential,
> >>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>> of this message is not the intended recipient, you are hereby notified
> that
> >>> any printing, copying, dissemination, distribution, disclosure or
> >>> forwarding of this communication is strictly prohibited. If you have
> >>> received this communication in error, please contact the sender
> immediately
> >>> and delete it from your system. Thank You.
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/hdp/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

>From YARN, we have a couple of items related to log-aggregation of services
and node-labels that are not tracked as blockers, but are working on
targeting 2.6. I am trying to get them in by end of this week.

+Vinod

On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> 2.6.0 is close now.
>
> Here are the remaining blockers, I'm hoping cut an RC in the next week or
> so:
> http://s.apache.org/hadoop-2.6.0-blockers
>
> thanks,
> Arun
>
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > Folks,
> >
> >  I've created branch-2.6 to stabilize the release.
> >
> >  Committers, please exercise caution henceforth on commits other than
> the ones we've discussed on this thread already.
> >
> >  By default new features should now be targeted to the version "2.7"
> henceforth - I've ensure all the projects have that version on jira.
> >
> > thanks,
> > Arun
> >
> > On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> >
> >> Sounds good. I'll branch this weekend and we can merge the jiras we
> >> discussed in this thread as they they get wrapped next week.
> >>
> >> Thanks everyone.
> >>
> >> Arun
> >>
> >>
> >>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >>>
> >>> We can branch off in a week or two so that work on branch-2 itself can
> go
> >>> ahead with other features that can't fit in 2.6. Independent of that,
> we
> >>> can then decide on the timeline of the release candidates once
> branch-2.6
> >>> is close to being done w.r.t the planned features.
> >>>
> >>> Branching it off can let us focus on specific features that we want in
> for
> >>> 2.6 and then eventually blockers for the release, nothing else. There
> is a
> >>> trivial pain of committing to one more branch, but it's worth it in
> this
> >>> case IMO.
> >>>
> >>> A lot of efforts are happening in parallel from the YARN side from
> where I
> >>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if
> we
> >>> don't branch off and selectively try to get stuff in, it is likely to
> be in
> >>> a perpetual delay.
> >>>
> >>> My 2 cents.
> >>>
> >>> +Vinod
> >>>
> >>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <
> suresh@hortonworks.com>
> >>> wrote:
> >>>
> >>>> Given some of the features are in final stages of stabilization,
> >>>> Arun, we should hold off creating 2.6 branch or building an RC by a
> week?
> >>>> All the features in flux are important ones and worth delaying the
> release
> >>>> by a week.
> >>>>
> >>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> Hey Nicholas,
> >>>>>
> >>>>> My concern about Archival Storage isn't related to the code quality
> or
> >>>> the
> >>>>> size of the feature. I think that you and Jing did good work. My
> concern
> >>>> is
> >>>>> that once we ship, we're locked into that set of archival storage
> APIs,
> >>>> and
> >>>>> these APIs are not yet finalized. Simply being able to turn off the
> >>>> feature
> >>>>> does not change the compatibility story.
> >>>>>
> >>>>> I'm willing to devote time to help review these JIRAs and kick the
> tires
> >>>> on
> >>>>> the APIs, but my point above was that I'm not sure it'd all be done
> by
> >>>> the
> >>>>> end of the week. Testing might also reveal additional changes that
> need
> >>>> to
> >>>>> be made, which also might not happen by end-of-week.
> >>>>>
> >>>>> I guess the question before us is if we're comfortable putting
> something
> >>>> in
> >>>>> branch-2.6 and then potentially adding API changes after. I'm okay
> with
> >>>>> that as long as we're all aware that this might happen.
> >>>>>
> >>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
> >>>> fine
> >>>>> with it's inclusion, just a heads up that we might need some extra
> time
> >>>> to
> >>>>> finalize things before an RC can be cut.
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>>
> >>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> >>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am worry about KMS and transparent encryption since there are
> quite
> >>>>> many
> >>>>>> bugs discovered after it got merged to branch-2.  It gives us an
> >>>>> impression
> >>>>>> that the feature is not yet well tested.  Indeed, transparent
> >>>> encryption
> >>>>> is
> >>>>>> a complicated feature which changes the core part of HDFS.  It is
> not
> >>>>> easy
> >>>>>> to get everything right.
> >>>>>>
> >>>>>>
> >>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low
> risk
> >>>>>> feature.  It introduces a new storage type ARCHIVE and the concept
> of
> >>>>> block
> >>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
> >>>>> storage,
> >>>>>> the blocks will be stored using the appropriate storage types
> specified
> >>>>> by
> >>>>>> storage policies assigned to the files/directories.  Cluster admin
> >>>> could
> >>>>>> disable the feature by simply not configuring any storage type and
> not
> >>>>>> setting any storage policy as before.   As Suresh mentioned,
> HDFS-6584
> >>>> is
> >>>>>> in the final stages to be merged to branch-2.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Tsz-Wo
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> >>>>>> suresh@hortonworks.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I actually would like to see both archival storage and single
> replica
> >>>>>>> memory writes to be in 2.6 release. Archival storage is in the
> final
> >>>>>> stages
> >>>>>>> of getting ready for branch-2 merge as Nicholas has already
> indicated
> >>>> on
> >>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both
> of
> >>>>> these
> >>>>>>> features are being in development for sometime.
> >>>>>>>
> >>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> >>>> andrew.wang@cloudera.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hey Arun,
> >>>>>>>>
> >>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
> >>>>>> add/retarget
> >>>>>>>> things accordingly?
> >>>>>>>>
> >>>>>>>> I think the KMS and transparent encryption are ready to go. We've
> >>>> got
> >>>>> a
> >>>>>>>> very few further bug fixes pending, but that's it.
> >>>>>>>>
> >>>>>>>> Two HDFS things that I think probably won't make the end of the
> week
> >>>>> are
> >>>>>>>> archival storage (HDFS-6584) and single replica memory writes
> >>>>>> (HDFS-6581),
> >>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just
> merged
> >>>> to
> >>>>>>>> trunk and I think needs a little more work before it goes into
> >>>>> branch-2.
> >>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> >>>> further
> >>>>>> off
> >>>>>>>> yet.
> >>>>>>>>
> >>>>>>>> Just my 2c as I did not work directly on these features. I just
> >>>>>> generally
> >>>>>>>> shy away from shipping bits quite this fresh.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <acm@hortonworks.com
> >
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
> >>>>>> nicely.
> >>>>>>>>>
> >>>>>>>>> I'll create branch-2.6 by end of the week and we can go from
> there
> >>>>> to
> >>>>>>>>> stabilize it - hopefully in the next few weeks.
> >>>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> >>>> acm@hortonworks.com
> >>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Folks,
> >>>>>>>>>>
> >>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
> >>>> to
> >>>>>>>>>> hadoop-2.6.
> >>>>>>>>>>
> >>>>>>>>>> Currently, here is the Roadmap per the wiki:
> >>>>>>>>>>
> >>>>>>>>>>       • HADOOP
> >>>>>>>>>>               • Credential provider HADOOP-10607
> >>>>>>>>>>       • HDFS
> >>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
> >>>>> for
> >>>>>>>>> using
> >>>>>>>>>> storage tiers by the applications HDFS-5682
> >>>>>>>>>>               • Memory as storage tier HDFS-5851
> >>>>>>>>>>       • YARN
> >>>>>>>>>>               • Dynamic Resource Configuration YARN-291
> >>>>>>>>>>               • NodeManager Restart YARN-1336
> >>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
> >>>>>>>>>>               • Support for admin-specified labels in YARN
> >>>>>> YARN-796
> >>>>>>>>>>               • Support for automatic, shared cache for YARN
> >>>>>>>>> application
> >>>>>>>>>> artifacts YARN-1492
> >>>>>>>>>>               • Support NodeGroup layer topology on YARN
> >>>> YARN-18
> >>>>>>>>>>               • Support for Docker containers in YARN
> >>>> YARN-1964
> >>>>>>>>>>               • YARN service registry YARN-913
> >>>>>>>>>>
> >>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
> >>>>>> won't.
> >>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
> >>>>>> would
> >>>>>>>> be
> >>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> >>>> up a
> >>>>>>>>> cadence.
> >>>>>>>>>>
> >>>>>>>>>> More importantly, as we discussed previously, we'd like
> >>>>> hadoop-2.6
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
> >>>>> start
> >>>>>> a
> >>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> >>>>> and
> >>>>>>>> see
> >>>>>>>>>> how they feel about this.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>> entity
> >>>>>>>> to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> the
> >>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> http://hortonworks.com/download/
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>>> to
> >>>>>>> which it is addressed and may contain information that is
> >>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://hortonworks.com/download/
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>>> of this message is not the intended recipient, you are hereby
> notified that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> immediately
> >>>> and delete it from your system. Thank You.
> >>>
> >>> --
> >>> CONFIDENTIALITY NOTICE
> >>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>> which it is addressed and may contain information that is confidential,
> >>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>> of this message is not the intended recipient, you are hereby notified
> that
> >>> any printing, copying, dissemination, distribution, disclosure or
> >>> forwarding of this communication is strictly prohibited. If you have
> >>> received this communication in error, please contact the sender
> immediately
> >>> and delete it from your system. Thank You.
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/hdp/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

>From YARN, we have a couple of items related to log-aggregation of services
and node-labels that are not tracked as blockers, but are working on
targeting 2.6. I am trying to get them in by end of this week.

+Vinod

On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> 2.6.0 is close now.
>
> Here are the remaining blockers, I'm hoping cut an RC in the next week or
> so:
> http://s.apache.org/hadoop-2.6.0-blockers
>
> thanks,
> Arun
>
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > Folks,
> >
> >  I've created branch-2.6 to stabilize the release.
> >
> >  Committers, please exercise caution henceforth on commits other than
> the ones we've discussed on this thread already.
> >
> >  By default new features should now be targeted to the version "2.7"
> henceforth - I've ensure all the projects have that version on jira.
> >
> > thanks,
> > Arun
> >
> > On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> >
> >> Sounds good. I'll branch this weekend and we can merge the jiras we
> >> discussed in this thread as they they get wrapped next week.
> >>
> >> Thanks everyone.
> >>
> >> Arun
> >>
> >>
> >>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >>>
> >>> We can branch off in a week or two so that work on branch-2 itself can
> go
> >>> ahead with other features that can't fit in 2.6. Independent of that,
> we
> >>> can then decide on the timeline of the release candidates once
> branch-2.6
> >>> is close to being done w.r.t the planned features.
> >>>
> >>> Branching it off can let us focus on specific features that we want in
> for
> >>> 2.6 and then eventually blockers for the release, nothing else. There
> is a
> >>> trivial pain of committing to one more branch, but it's worth it in
> this
> >>> case IMO.
> >>>
> >>> A lot of efforts are happening in parallel from the YARN side from
> where I
> >>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if
> we
> >>> don't branch off and selectively try to get stuff in, it is likely to
> be in
> >>> a perpetual delay.
> >>>
> >>> My 2 cents.
> >>>
> >>> +Vinod
> >>>
> >>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <
> suresh@hortonworks.com>
> >>> wrote:
> >>>
> >>>> Given some of the features are in final stages of stabilization,
> >>>> Arun, we should hold off creating 2.6 branch or building an RC by a
> week?
> >>>> All the features in flux are important ones and worth delaying the
> release
> >>>> by a week.
> >>>>
> >>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> Hey Nicholas,
> >>>>>
> >>>>> My concern about Archival Storage isn't related to the code quality
> or
> >>>> the
> >>>>> size of the feature. I think that you and Jing did good work. My
> concern
> >>>> is
> >>>>> that once we ship, we're locked into that set of archival storage
> APIs,
> >>>> and
> >>>>> these APIs are not yet finalized. Simply being able to turn off the
> >>>> feature
> >>>>> does not change the compatibility story.
> >>>>>
> >>>>> I'm willing to devote time to help review these JIRAs and kick the
> tires
> >>>> on
> >>>>> the APIs, but my point above was that I'm not sure it'd all be done
> by
> >>>> the
> >>>>> end of the week. Testing might also reveal additional changes that
> need
> >>>> to
> >>>>> be made, which also might not happen by end-of-week.
> >>>>>
> >>>>> I guess the question before us is if we're comfortable putting
> something
> >>>> in
> >>>>> branch-2.6 and then potentially adding API changes after. I'm okay
> with
> >>>>> that as long as we're all aware that this might happen.
> >>>>>
> >>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
> >>>> fine
> >>>>> with it's inclusion, just a heads up that we might need some extra
> time
> >>>> to
> >>>>> finalize things before an RC can be cut.
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>>
> >>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> >>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am worry about KMS and transparent encryption since there are
> quite
> >>>>> many
> >>>>>> bugs discovered after it got merged to branch-2.  It gives us an
> >>>>> impression
> >>>>>> that the feature is not yet well tested.  Indeed, transparent
> >>>> encryption
> >>>>> is
> >>>>>> a complicated feature which changes the core part of HDFS.  It is
> not
> >>>>> easy
> >>>>>> to get everything right.
> >>>>>>
> >>>>>>
> >>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low
> risk
> >>>>>> feature.  It introduces a new storage type ARCHIVE and the concept
> of
> >>>>> block
> >>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
> >>>>> storage,
> >>>>>> the blocks will be stored using the appropriate storage types
> specified
> >>>>> by
> >>>>>> storage policies assigned to the files/directories.  Cluster admin
> >>>> could
> >>>>>> disable the feature by simply not configuring any storage type and
> not
> >>>>>> setting any storage policy as before.   As Suresh mentioned,
> HDFS-6584
> >>>> is
> >>>>>> in the final stages to be merged to branch-2.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Tsz-Wo
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> >>>>>> suresh@hortonworks.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I actually would like to see both archival storage and single
> replica
> >>>>>>> memory writes to be in 2.6 release. Archival storage is in the
> final
> >>>>>> stages
> >>>>>>> of getting ready for branch-2 merge as Nicholas has already
> indicated
> >>>> on
> >>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both
> of
> >>>>> these
> >>>>>>> features are being in development for sometime.
> >>>>>>>
> >>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> >>>> andrew.wang@cloudera.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hey Arun,
> >>>>>>>>
> >>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
> >>>>>> add/retarget
> >>>>>>>> things accordingly?
> >>>>>>>>
> >>>>>>>> I think the KMS and transparent encryption are ready to go. We've
> >>>> got
> >>>>> a
> >>>>>>>> very few further bug fixes pending, but that's it.
> >>>>>>>>
> >>>>>>>> Two HDFS things that I think probably won't make the end of the
> week
> >>>>> are
> >>>>>>>> archival storage (HDFS-6584) and single replica memory writes
> >>>>>> (HDFS-6581),
> >>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just
> merged
> >>>> to
> >>>>>>>> trunk and I think needs a little more work before it goes into
> >>>>> branch-2.
> >>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> >>>> further
> >>>>>> off
> >>>>>>>> yet.
> >>>>>>>>
> >>>>>>>> Just my 2c as I did not work directly on these features. I just
> >>>>>> generally
> >>>>>>>> shy away from shipping bits quite this fresh.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <acm@hortonworks.com
> >
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
> >>>>>> nicely.
> >>>>>>>>>
> >>>>>>>>> I'll create branch-2.6 by end of the week and we can go from
> there
> >>>>> to
> >>>>>>>>> stabilize it - hopefully in the next few weeks.
> >>>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> >>>> acm@hortonworks.com
> >>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Folks,
> >>>>>>>>>>
> >>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
> >>>> to
> >>>>>>>>>> hadoop-2.6.
> >>>>>>>>>>
> >>>>>>>>>> Currently, here is the Roadmap per the wiki:
> >>>>>>>>>>
> >>>>>>>>>>       • HADOOP
> >>>>>>>>>>               • Credential provider HADOOP-10607
> >>>>>>>>>>       • HDFS
> >>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
> >>>>> for
> >>>>>>>>> using
> >>>>>>>>>> storage tiers by the applications HDFS-5682
> >>>>>>>>>>               • Memory as storage tier HDFS-5851
> >>>>>>>>>>       • YARN
> >>>>>>>>>>               • Dynamic Resource Configuration YARN-291
> >>>>>>>>>>               • NodeManager Restart YARN-1336
> >>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
> >>>>>>>>>>               • Support for admin-specified labels in YARN
> >>>>>> YARN-796
> >>>>>>>>>>               • Support for automatic, shared cache for YARN
> >>>>>>>>> application
> >>>>>>>>>> artifacts YARN-1492
> >>>>>>>>>>               • Support NodeGroup layer topology on YARN
> >>>> YARN-18
> >>>>>>>>>>               • Support for Docker containers in YARN
> >>>> YARN-1964
> >>>>>>>>>>               • YARN service registry YARN-913
> >>>>>>>>>>
> >>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
> >>>>>> won't.
> >>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
> >>>>>> would
> >>>>>>>> be
> >>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> >>>> up a
> >>>>>>>>> cadence.
> >>>>>>>>>>
> >>>>>>>>>> More importantly, as we discussed previously, we'd like
> >>>>> hadoop-2.6
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
> >>>>> start
> >>>>>> a
> >>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> >>>>> and
> >>>>>>>> see
> >>>>>>>>>> how they feel about this.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>> entity
> >>>>>>>> to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> the
> >>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> http://hortonworks.com/download/
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>>> to
> >>>>>>> which it is addressed and may contain information that is
> >>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://hortonworks.com/download/
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>>> of this message is not the intended recipient, you are hereby
> notified that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> immediately
> >>>> and delete it from your system. Thank You.
> >>>
> >>> --
> >>> CONFIDENTIALITY NOTICE
> >>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>> which it is addressed and may contain information that is confidential,
> >>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>> of this message is not the intended recipient, you are hereby notified
> that
> >>> any printing, copying, dissemination, distribution, disclosure or
> >>> forwarding of this communication is strictly prohibited. If you have
> >>> received this communication in error, please contact the sender
> immediately
> >>> and delete it from your system. Thank You.
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/hdp/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

Looks like most of the big ones are in. I'll try prod people into finishing up the rest and get an RC going this week.

Thanks everyone!

Arun

On Oct 14, 2014, at 11:25 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> 2.6.0 is close now.
> 
> Here are the remaining blockers, I'm hoping cut an RC in the next week or so:
> http://s.apache.org/hadoop-2.6.0-blockers
> 
> thanks,
> Arun
> 
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> Folks,
>> 
>>  I've created branch-2.6 to stabilize the release.
>>  
>>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
>> 
>>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
>> 
>> thanks,
>> Arun
>> 
>> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
>> 
>>> Sounds good. I'll branch this weekend and we can merge the jiras we
>>> discussed in this thread as they they get wrapped next week.
>>> 
>>> Thanks everyone.
>>> 
>>> Arun
>>> 
>>> 
>>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>> 
>>>> We can branch off in a week or two so that work on branch-2 itself can go
>>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>>> can then decide on the timeline of the release candidates once branch-2.6
>>>> is close to being done w.r.t the planned features.
>>>> 
>>>> Branching it off can let us focus on specific features that we want in for
>>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>>> trivial pain of committing to one more branch, but it's worth it in this
>>>> case IMO.
>>>> 
>>>> A lot of efforts are happening in parallel from the YARN side from where I
>>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>>> a perpetual delay.
>>>> 
>>>> My 2 cents.
>>>> 
>>>> +Vinod
>>>> 
>>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>>> wrote:
>>>> 
>>>>> Given some of the features are in final stages of stabilization,
>>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>>> All the features in flux are important ones and worth delaying the release
>>>>> by a week.
>>>>> 
>>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>>> wrote:
>>>>> 
>>>>>> Hey Nicholas,
>>>>>> 
>>>>>> My concern about Archival Storage isn't related to the code quality or
>>>>> the
>>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>>> is
>>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>>> and
>>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>>> feature
>>>>>> does not change the compatibility story.
>>>>>> 
>>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>>> on
>>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>>> the
>>>>>> end of the week. Testing might also reveal additional changes that need
>>>>> to
>>>>>> be made, which also might not happen by end-of-week.
>>>>>> 
>>>>>> I guess the question before us is if we're comfortable putting something
>>>>> in
>>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>>> that as long as we're all aware that this might happen.
>>>>>> 
>>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>>> fine
>>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>>> to
>>>>>> finalize things before an RC can be cut.
>>>>>> 
>>>>>> Thanks,
>>>>>> Andrew
>>>>>> 
>>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>>> many
>>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>>> impression
>>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>>> encryption
>>>>>> is
>>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>>> easy
>>>>>>> to get everything right.
>>>>>>> 
>>>>>>> 
>>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>>> block
>>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>>> storage,
>>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>>> by
>>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>>> could
>>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>>> is
>>>>>>> in the final stages to be merged to branch-2.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Tsz-Wo
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>>> suresh@hortonworks.com> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>>> stages
>>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>>> on
>>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>>> these
>>>>>>>> features are being in development for sometime.
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>>> andrew.wang@cloudera.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey Arun,
>>>>>>>>> 
>>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>>> add/retarget
>>>>>>>>> things accordingly?
>>>>>>>>> 
>>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>>> got
>>>>>> a
>>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>>> 
>>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>>> are
>>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>>> (HDFS-6581),
>>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>>> to
>>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>>> branch-2.
>>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>>> further
>>>>>>> off
>>>>>>>>> yet.
>>>>>>>>> 
>>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>>> generally
>>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
>>>>>>>>> 
>>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>>> nicely.
>>>>>>>>>> 
>>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>>> to
>>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>>> 
>>>>>>>>>> Thoughts?
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>> 
>>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>>> acm@hortonworks.com
>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Folks,
>>>>>>>>>>> 
>>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>>> to
>>>>>>>>>>> hadoop-2.6.
>>>>>>>>>>> 
>>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>>> 
>>>>>>>>>>>       • HADOOP
>>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>>       • HDFS
>>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>>> for
>>>>>>>>>> using
>>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>>       • YARN
>>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>>> YARN-796
>>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>>> application
>>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>>> YARN-18
>>>>>>>>>>>               • Support for Docker containers in YARN
>>>>> YARN-1964
>>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>>> 
>>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>>> won't.
>>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>>> would
>>>>>>>>> be
>>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>>> up a
>>>>>>>>>> cadence.
>>>>>>>>>>> 
>>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>>> hadoop-2.6
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>>> start
>>>>>>> a
>>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>>> and
>>>>>>>>> see
>>>>>>>>>>> how they feel about this.
>>>>>>>>>>> 
>>>>>>>>>>> thanks,
>>>>>>>>>>> Arun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Arun C. Murthy
>>>>>>>>>> Hortonworks Inc.
>>>>>>>>>> http://hortonworks.com/
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>> entity
>>>>>>>>> to
>>>>>>>>>> which it is addressed and may contain information that is
>>>>>>> confidential,
>>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>> reader
>>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>>> notified
>>>>>>>>> that
>>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>>> have
>>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>>> immediately
>>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> http://hortonworks.com/download/
>>>>>>>> 
>>>>>>>> --
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity
>>>>>>> to
>>>>>>>> which it is addressed and may contain information that is
>>>>> confidential,
>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>>> that
>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>>> received this communication in error, please contact the sender
>>>>>>> immediately
>>>>>>>> and delete it from your system. Thank You.
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>> 
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>>> of this message is not the intended recipient, you are hereby notified that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender immediately
>>>>> and delete it from your system. Thank You.
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/hdp/
>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

>From YARN, we have a couple of items related to log-aggregation of services
and node-labels that are not tracked as blockers, but are working on
targeting 2.6. I am trying to get them in by end of this week.

+Vinod

On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> 2.6.0 is close now.
>
> Here are the remaining blockers, I'm hoping cut an RC in the next week or
> so:
> http://s.apache.org/hadoop-2.6.0-blockers
>
> thanks,
> Arun
>
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > Folks,
> >
> >  I've created branch-2.6 to stabilize the release.
> >
> >  Committers, please exercise caution henceforth on commits other than
> the ones we've discussed on this thread already.
> >
> >  By default new features should now be targeted to the version "2.7"
> henceforth - I've ensure all the projects have that version on jira.
> >
> > thanks,
> > Arun
> >
> > On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> >
> >> Sounds good. I'll branch this weekend and we can merge the jiras we
> >> discussed in this thread as they they get wrapped next week.
> >>
> >> Thanks everyone.
> >>
> >> Arun
> >>
> >>
> >>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >>>
> >>> We can branch off in a week or two so that work on branch-2 itself can
> go
> >>> ahead with other features that can't fit in 2.6. Independent of that,
> we
> >>> can then decide on the timeline of the release candidates once
> branch-2.6
> >>> is close to being done w.r.t the planned features.
> >>>
> >>> Branching it off can let us focus on specific features that we want in
> for
> >>> 2.6 and then eventually blockers for the release, nothing else. There
> is a
> >>> trivial pain of committing to one more branch, but it's worth it in
> this
> >>> case IMO.
> >>>
> >>> A lot of efforts are happening in parallel from the YARN side from
> where I
> >>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if
> we
> >>> don't branch off and selectively try to get stuff in, it is likely to
> be in
> >>> a perpetual delay.
> >>>
> >>> My 2 cents.
> >>>
> >>> +Vinod
> >>>
> >>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <
> suresh@hortonworks.com>
> >>> wrote:
> >>>
> >>>> Given some of the features are in final stages of stabilization,
> >>>> Arun, we should hold off creating 2.6 branch or building an RC by a
> week?
> >>>> All the features in flux are important ones and worth delaying the
> release
> >>>> by a week.
> >>>>
> >>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> Hey Nicholas,
> >>>>>
> >>>>> My concern about Archival Storage isn't related to the code quality
> or
> >>>> the
> >>>>> size of the feature. I think that you and Jing did good work. My
> concern
> >>>> is
> >>>>> that once we ship, we're locked into that set of archival storage
> APIs,
> >>>> and
> >>>>> these APIs are not yet finalized. Simply being able to turn off the
> >>>> feature
> >>>>> does not change the compatibility story.
> >>>>>
> >>>>> I'm willing to devote time to help review these JIRAs and kick the
> tires
> >>>> on
> >>>>> the APIs, but my point above was that I'm not sure it'd all be done
> by
> >>>> the
> >>>>> end of the week. Testing might also reveal additional changes that
> need
> >>>> to
> >>>>> be made, which also might not happen by end-of-week.
> >>>>>
> >>>>> I guess the question before us is if we're comfortable putting
> something
> >>>> in
> >>>>> branch-2.6 and then potentially adding API changes after. I'm okay
> with
> >>>>> that as long as we're all aware that this might happen.
> >>>>>
> >>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
> >>>> fine
> >>>>> with it's inclusion, just a heads up that we might need some extra
> time
> >>>> to
> >>>>> finalize things before an RC can be cut.
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>>
> >>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> >>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am worry about KMS and transparent encryption since there are
> quite
> >>>>> many
> >>>>>> bugs discovered after it got merged to branch-2.  It gives us an
> >>>>> impression
> >>>>>> that the feature is not yet well tested.  Indeed, transparent
> >>>> encryption
> >>>>> is
> >>>>>> a complicated feature which changes the core part of HDFS.  It is
> not
> >>>>> easy
> >>>>>> to get everything right.
> >>>>>>
> >>>>>>
> >>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low
> risk
> >>>>>> feature.  It introduces a new storage type ARCHIVE and the concept
> of
> >>>>> block
> >>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
> >>>>> storage,
> >>>>>> the blocks will be stored using the appropriate storage types
> specified
> >>>>> by
> >>>>>> storage policies assigned to the files/directories.  Cluster admin
> >>>> could
> >>>>>> disable the feature by simply not configuring any storage type and
> not
> >>>>>> setting any storage policy as before.   As Suresh mentioned,
> HDFS-6584
> >>>> is
> >>>>>> in the final stages to be merged to branch-2.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Tsz-Wo
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> >>>>>> suresh@hortonworks.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I actually would like to see both archival storage and single
> replica
> >>>>>>> memory writes to be in 2.6 release. Archival storage is in the
> final
> >>>>>> stages
> >>>>>>> of getting ready for branch-2 merge as Nicholas has already
> indicated
> >>>> on
> >>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both
> of
> >>>>> these
> >>>>>>> features are being in development for sometime.
> >>>>>>>
> >>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> >>>> andrew.wang@cloudera.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hey Arun,
> >>>>>>>>
> >>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
> >>>>>> add/retarget
> >>>>>>>> things accordingly?
> >>>>>>>>
> >>>>>>>> I think the KMS and transparent encryption are ready to go. We've
> >>>> got
> >>>>> a
> >>>>>>>> very few further bug fixes pending, but that's it.
> >>>>>>>>
> >>>>>>>> Two HDFS things that I think probably won't make the end of the
> week
> >>>>> are
> >>>>>>>> archival storage (HDFS-6584) and single replica memory writes
> >>>>>> (HDFS-6581),
> >>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just
> merged
> >>>> to
> >>>>>>>> trunk and I think needs a little more work before it goes into
> >>>>> branch-2.
> >>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> >>>> further
> >>>>>> off
> >>>>>>>> yet.
> >>>>>>>>
> >>>>>>>> Just my 2c as I did not work directly on these features. I just
> >>>>>> generally
> >>>>>>>> shy away from shipping bits quite this fresh.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <acm@hortonworks.com
> >
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
> >>>>>> nicely.
> >>>>>>>>>
> >>>>>>>>> I'll create branch-2.6 by end of the week and we can go from
> there
> >>>>> to
> >>>>>>>>> stabilize it - hopefully in the next few weeks.
> >>>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> >>>> acm@hortonworks.com
> >>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Folks,
> >>>>>>>>>>
> >>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
> >>>> to
> >>>>>>>>>> hadoop-2.6.
> >>>>>>>>>>
> >>>>>>>>>> Currently, here is the Roadmap per the wiki:
> >>>>>>>>>>
> >>>>>>>>>>       • HADOOP
> >>>>>>>>>>               • Credential provider HADOOP-10607
> >>>>>>>>>>       • HDFS
> >>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
> >>>>> for
> >>>>>>>>> using
> >>>>>>>>>> storage tiers by the applications HDFS-5682
> >>>>>>>>>>               • Memory as storage tier HDFS-5851
> >>>>>>>>>>       • YARN
> >>>>>>>>>>               • Dynamic Resource Configuration YARN-291
> >>>>>>>>>>               • NodeManager Restart YARN-1336
> >>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
> >>>>>>>>>>               • Support for admin-specified labels in YARN
> >>>>>> YARN-796
> >>>>>>>>>>               • Support for automatic, shared cache for YARN
> >>>>>>>>> application
> >>>>>>>>>> artifacts YARN-1492
> >>>>>>>>>>               • Support NodeGroup layer topology on YARN
> >>>> YARN-18
> >>>>>>>>>>               • Support for Docker containers in YARN
> >>>> YARN-1964
> >>>>>>>>>>               • YARN service registry YARN-913
> >>>>>>>>>>
> >>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
> >>>>>> won't.
> >>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
> >>>>>> would
> >>>>>>>> be
> >>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> >>>> up a
> >>>>>>>>> cadence.
> >>>>>>>>>>
> >>>>>>>>>> More importantly, as we discussed previously, we'd like
> >>>>> hadoop-2.6
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
> >>>>> start
> >>>>>> a
> >>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> >>>>> and
> >>>>>>>> see
> >>>>>>>>>> how they feel about this.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>> entity
> >>>>>>>> to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> the
> >>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> http://hortonworks.com/download/
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>>> to
> >>>>>>> which it is addressed and may contain information that is
> >>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://hortonworks.com/download/
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>>> of this message is not the intended recipient, you are hereby
> notified that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> immediately
> >>>> and delete it from your system. Thank You.
> >>>
> >>> --
> >>> CONFIDENTIALITY NOTICE
> >>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>> which it is addressed and may contain information that is confidential,
> >>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>> of this message is not the intended recipient, you are hereby notified
> that
> >>> any printing, copying, dissemination, distribution, disclosure or
> >>> forwarding of this communication is strictly prohibited. If you have
> >>> received this communication in error, please contact the sender
> immediately
> >>> and delete it from your system. Thank You.
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/hdp/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

Looks like most of the big ones are in. I'll try prod people into finishing up the rest and get an RC going this week.

Thanks everyone!

Arun

On Oct 14, 2014, at 11:25 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> 2.6.0 is close now.
> 
> Here are the remaining blockers, I'm hoping cut an RC in the next week or so:
> http://s.apache.org/hadoop-2.6.0-blockers
> 
> thanks,
> Arun
> 
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> Folks,
>> 
>>  I've created branch-2.6 to stabilize the release.
>>  
>>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
>> 
>>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
>> 
>> thanks,
>> Arun
>> 
>> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
>> 
>>> Sounds good. I'll branch this weekend and we can merge the jiras we
>>> discussed in this thread as they they get wrapped next week.
>>> 
>>> Thanks everyone.
>>> 
>>> Arun
>>> 
>>> 
>>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>> 
>>>> We can branch off in a week or two so that work on branch-2 itself can go
>>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>>> can then decide on the timeline of the release candidates once branch-2.6
>>>> is close to being done w.r.t the planned features.
>>>> 
>>>> Branching it off can let us focus on specific features that we want in for
>>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>>> trivial pain of committing to one more branch, but it's worth it in this
>>>> case IMO.
>>>> 
>>>> A lot of efforts are happening in parallel from the YARN side from where I
>>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>>> a perpetual delay.
>>>> 
>>>> My 2 cents.
>>>> 
>>>> +Vinod
>>>> 
>>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>>> wrote:
>>>> 
>>>>> Given some of the features are in final stages of stabilization,
>>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>>> All the features in flux are important ones and worth delaying the release
>>>>> by a week.
>>>>> 
>>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>>> wrote:
>>>>> 
>>>>>> Hey Nicholas,
>>>>>> 
>>>>>> My concern about Archival Storage isn't related to the code quality or
>>>>> the
>>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>>> is
>>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>>> and
>>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>>> feature
>>>>>> does not change the compatibility story.
>>>>>> 
>>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>>> on
>>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>>> the
>>>>>> end of the week. Testing might also reveal additional changes that need
>>>>> to
>>>>>> be made, which also might not happen by end-of-week.
>>>>>> 
>>>>>> I guess the question before us is if we're comfortable putting something
>>>>> in
>>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>>> that as long as we're all aware that this might happen.
>>>>>> 
>>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>>> fine
>>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>>> to
>>>>>> finalize things before an RC can be cut.
>>>>>> 
>>>>>> Thanks,
>>>>>> Andrew
>>>>>> 
>>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>>> many
>>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>>> impression
>>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>>> encryption
>>>>>> is
>>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>>> easy
>>>>>>> to get everything right.
>>>>>>> 
>>>>>>> 
>>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>>> block
>>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>>> storage,
>>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>>> by
>>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>>> could
>>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>>> is
>>>>>>> in the final stages to be merged to branch-2.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Tsz-Wo
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>>> suresh@hortonworks.com> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>>> stages
>>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>>> on
>>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>>> these
>>>>>>>> features are being in development for sometime.
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>>> andrew.wang@cloudera.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey Arun,
>>>>>>>>> 
>>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>>> add/retarget
>>>>>>>>> things accordingly?
>>>>>>>>> 
>>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>>> got
>>>>>> a
>>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>>> 
>>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>>> are
>>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>>> (HDFS-6581),
>>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>>> to
>>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>>> branch-2.
>>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>>> further
>>>>>>> off
>>>>>>>>> yet.
>>>>>>>>> 
>>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>>> generally
>>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
>>>>>>>>> 
>>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>>> nicely.
>>>>>>>>>> 
>>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>>> to
>>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>>> 
>>>>>>>>>> Thoughts?
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>> 
>>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>>> acm@hortonworks.com
>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Folks,
>>>>>>>>>>> 
>>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>>> to
>>>>>>>>>>> hadoop-2.6.
>>>>>>>>>>> 
>>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>>> 
>>>>>>>>>>>       • HADOOP
>>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>>       • HDFS
>>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>>> for
>>>>>>>>>> using
>>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>>       • YARN
>>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>>> YARN-796
>>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>>> application
>>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>>> YARN-18
>>>>>>>>>>>               • Support for Docker containers in YARN
>>>>> YARN-1964
>>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>>> 
>>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>>> won't.
>>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>>> would
>>>>>>>>> be
>>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>>> up a
>>>>>>>>>> cadence.
>>>>>>>>>>> 
>>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>>> hadoop-2.6
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>>> start
>>>>>>> a
>>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>>> and
>>>>>>>>> see
>>>>>>>>>>> how they feel about this.
>>>>>>>>>>> 
>>>>>>>>>>> thanks,
>>>>>>>>>>> Arun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Arun C. Murthy
>>>>>>>>>> Hortonworks Inc.
>>>>>>>>>> http://hortonworks.com/
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>> entity
>>>>>>>>> to
>>>>>>>>>> which it is addressed and may contain information that is
>>>>>>> confidential,
>>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>> reader
>>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>>> notified
>>>>>>>>> that
>>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>>> have
>>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>>> immediately
>>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> http://hortonworks.com/download/
>>>>>>>> 
>>>>>>>> --
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity
>>>>>>> to
>>>>>>>> which it is addressed and may contain information that is
>>>>> confidential,
>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>>> that
>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>>> received this communication in error, please contact the sender
>>>>>>> immediately
>>>>>>>> and delete it from your system. Thank You.
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>> 
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>>> of this message is not the intended recipient, you are hereby notified that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender immediately
>>>>> and delete it from your system. Thank You.
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/hdp/
>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

Looks like most of the big ones are in. I'll try prod people into finishing up the rest and get an RC going this week.

Thanks everyone!

Arun

On Oct 14, 2014, at 11:25 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> 2.6.0 is close now.
> 
> Here are the remaining blockers, I'm hoping cut an RC in the next week or so:
> http://s.apache.org/hadoop-2.6.0-blockers
> 
> thanks,
> Arun
> 
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> Folks,
>> 
>>  I've created branch-2.6 to stabilize the release.
>>  
>>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
>> 
>>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
>> 
>> thanks,
>> Arun
>> 
>> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
>> 
>>> Sounds good. I'll branch this weekend and we can merge the jiras we
>>> discussed in this thread as they they get wrapped next week.
>>> 
>>> Thanks everyone.
>>> 
>>> Arun
>>> 
>>> 
>>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>> 
>>>> We can branch off in a week or two so that work on branch-2 itself can go
>>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>>> can then decide on the timeline of the release candidates once branch-2.6
>>>> is close to being done w.r.t the planned features.
>>>> 
>>>> Branching it off can let us focus on specific features that we want in for
>>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>>> trivial pain of committing to one more branch, but it's worth it in this
>>>> case IMO.
>>>> 
>>>> A lot of efforts are happening in parallel from the YARN side from where I
>>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>>> a perpetual delay.
>>>> 
>>>> My 2 cents.
>>>> 
>>>> +Vinod
>>>> 
>>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>>> wrote:
>>>> 
>>>>> Given some of the features are in final stages of stabilization,
>>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>>> All the features in flux are important ones and worth delaying the release
>>>>> by a week.
>>>>> 
>>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>>> wrote:
>>>>> 
>>>>>> Hey Nicholas,
>>>>>> 
>>>>>> My concern about Archival Storage isn't related to the code quality or
>>>>> the
>>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>>> is
>>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>>> and
>>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>>> feature
>>>>>> does not change the compatibility story.
>>>>>> 
>>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>>> on
>>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>>> the
>>>>>> end of the week. Testing might also reveal additional changes that need
>>>>> to
>>>>>> be made, which also might not happen by end-of-week.
>>>>>> 
>>>>>> I guess the question before us is if we're comfortable putting something
>>>>> in
>>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>>> that as long as we're all aware that this might happen.
>>>>>> 
>>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>>> fine
>>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>>> to
>>>>>> finalize things before an RC can be cut.
>>>>>> 
>>>>>> Thanks,
>>>>>> Andrew
>>>>>> 
>>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>>> many
>>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>>> impression
>>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>>> encryption
>>>>>> is
>>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>>> easy
>>>>>>> to get everything right.
>>>>>>> 
>>>>>>> 
>>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>>> block
>>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>>> storage,
>>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>>> by
>>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>>> could
>>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>>> is
>>>>>>> in the final stages to be merged to branch-2.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Tsz-Wo
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>>> suresh@hortonworks.com> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>>> stages
>>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>>> on
>>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>>> these
>>>>>>>> features are being in development for sometime.
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>>> andrew.wang@cloudera.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey Arun,
>>>>>>>>> 
>>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>>> add/retarget
>>>>>>>>> things accordingly?
>>>>>>>>> 
>>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>>> got
>>>>>> a
>>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>>> 
>>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>>> are
>>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>>> (HDFS-6581),
>>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>>> to
>>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>>> branch-2.
>>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>>> further
>>>>>>> off
>>>>>>>>> yet.
>>>>>>>>> 
>>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>>> generally
>>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
>>>>>>>>> 
>>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>>> nicely.
>>>>>>>>>> 
>>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>>> to
>>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>>> 
>>>>>>>>>> Thoughts?
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>> 
>>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>>> acm@hortonworks.com
>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Folks,
>>>>>>>>>>> 
>>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>>> to
>>>>>>>>>>> hadoop-2.6.
>>>>>>>>>>> 
>>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>>> 
>>>>>>>>>>>       • HADOOP
>>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>>       • HDFS
>>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>>> for
>>>>>>>>>> using
>>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>>       • YARN
>>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>>> YARN-796
>>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>>> application
>>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>>> YARN-18
>>>>>>>>>>>               • Support for Docker containers in YARN
>>>>> YARN-1964
>>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>>> 
>>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>>> won't.
>>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>>> would
>>>>>>>>> be
>>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>>> up a
>>>>>>>>>> cadence.
>>>>>>>>>>> 
>>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>>> hadoop-2.6
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>>> start
>>>>>>> a
>>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>>> and
>>>>>>>>> see
>>>>>>>>>>> how they feel about this.
>>>>>>>>>>> 
>>>>>>>>>>> thanks,
>>>>>>>>>>> Arun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Arun C. Murthy
>>>>>>>>>> Hortonworks Inc.
>>>>>>>>>> http://hortonworks.com/
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>> entity
>>>>>>>>> to
>>>>>>>>>> which it is addressed and may contain information that is
>>>>>>> confidential,
>>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>> reader
>>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>>> notified
>>>>>>>>> that
>>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>>> have
>>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>>> immediately
>>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> http://hortonworks.com/download/
>>>>>>>> 
>>>>>>>> --
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity
>>>>>>> to
>>>>>>>> which it is addressed and may contain information that is
>>>>> confidential,
>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>>> that
>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>>> received this communication in error, please contact the sender
>>>>>>> immediately
>>>>>>>> and delete it from your system. Thank You.
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>> 
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>>> of this message is not the intended recipient, you are hereby notified that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender immediately
>>>>> and delete it from your system. Thank You.
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/hdp/
>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

Looks like most of the big ones are in. I'll try prod people into finishing up the rest and get an RC going this week.

Thanks everyone!

Arun

On Oct 14, 2014, at 11:25 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> 2.6.0 is close now.
> 
> Here are the remaining blockers, I'm hoping cut an RC in the next week or so:
> http://s.apache.org/hadoop-2.6.0-blockers
> 
> thanks,
> Arun
> 
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> Folks,
>> 
>>  I've created branch-2.6 to stabilize the release.
>>  
>>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
>> 
>>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
>> 
>> thanks,
>> Arun
>> 
>> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
>> 
>>> Sounds good. I'll branch this weekend and we can merge the jiras we
>>> discussed in this thread as they they get wrapped next week.
>>> 
>>> Thanks everyone.
>>> 
>>> Arun
>>> 
>>> 
>>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>> 
>>>> We can branch off in a week or two so that work on branch-2 itself can go
>>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>>> can then decide on the timeline of the release candidates once branch-2.6
>>>> is close to being done w.r.t the planned features.
>>>> 
>>>> Branching it off can let us focus on specific features that we want in for
>>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>>> trivial pain of committing to one more branch, but it's worth it in this
>>>> case IMO.
>>>> 
>>>> A lot of efforts are happening in parallel from the YARN side from where I
>>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>>> a perpetual delay.
>>>> 
>>>> My 2 cents.
>>>> 
>>>> +Vinod
>>>> 
>>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>>> wrote:
>>>> 
>>>>> Given some of the features are in final stages of stabilization,
>>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>>> All the features in flux are important ones and worth delaying the release
>>>>> by a week.
>>>>> 
>>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>>> wrote:
>>>>> 
>>>>>> Hey Nicholas,
>>>>>> 
>>>>>> My concern about Archival Storage isn't related to the code quality or
>>>>> the
>>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>>> is
>>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>>> and
>>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>>> feature
>>>>>> does not change the compatibility story.
>>>>>> 
>>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>>> on
>>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>>> the
>>>>>> end of the week. Testing might also reveal additional changes that need
>>>>> to
>>>>>> be made, which also might not happen by end-of-week.
>>>>>> 
>>>>>> I guess the question before us is if we're comfortable putting something
>>>>> in
>>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>>> that as long as we're all aware that this might happen.
>>>>>> 
>>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>>> fine
>>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>>> to
>>>>>> finalize things before an RC can be cut.
>>>>>> 
>>>>>> Thanks,
>>>>>> Andrew
>>>>>> 
>>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>>> many
>>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>>> impression
>>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>>> encryption
>>>>>> is
>>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>>> easy
>>>>>>> to get everything right.
>>>>>>> 
>>>>>>> 
>>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>>> block
>>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>>> storage,
>>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>>> by
>>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>>> could
>>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>>> is
>>>>>>> in the final stages to be merged to branch-2.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Tsz-Wo
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>>> suresh@hortonworks.com> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>>> stages
>>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>>> on
>>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>>> these
>>>>>>>> features are being in development for sometime.
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>>> andrew.wang@cloudera.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey Arun,
>>>>>>>>> 
>>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>>> add/retarget
>>>>>>>>> things accordingly?
>>>>>>>>> 
>>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>>> got
>>>>>> a
>>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>>> 
>>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>>> are
>>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>>> (HDFS-6581),
>>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>>> to
>>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>>> branch-2.
>>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>>> further
>>>>>>> off
>>>>>>>>> yet.
>>>>>>>>> 
>>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>>> generally
>>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
>>>>>>>>> 
>>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>>> nicely.
>>>>>>>>>> 
>>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>>> to
>>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>>> 
>>>>>>>>>> Thoughts?
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>> 
>>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>>> acm@hortonworks.com
>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Folks,
>>>>>>>>>>> 
>>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>>> to
>>>>>>>>>>> hadoop-2.6.
>>>>>>>>>>> 
>>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>>> 
>>>>>>>>>>>       • HADOOP
>>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>>       • HDFS
>>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>>> for
>>>>>>>>>> using
>>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>>       • YARN
>>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>>> YARN-796
>>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>>> application
>>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>>> YARN-18
>>>>>>>>>>>               • Support for Docker containers in YARN
>>>>> YARN-1964
>>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>>> 
>>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>>> won't.
>>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>>> would
>>>>>>>>> be
>>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>>> up a
>>>>>>>>>> cadence.
>>>>>>>>>>> 
>>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>>> hadoop-2.6
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>>> start
>>>>>>> a
>>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>>> and
>>>>>>>>> see
>>>>>>>>>>> how they feel about this.
>>>>>>>>>>> 
>>>>>>>>>>> thanks,
>>>>>>>>>>> Arun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Arun C. Murthy
>>>>>>>>>> Hortonworks Inc.
>>>>>>>>>> http://hortonworks.com/
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>> entity
>>>>>>>>> to
>>>>>>>>>> which it is addressed and may contain information that is
>>>>>>> confidential,
>>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>> reader
>>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>>> notified
>>>>>>>>> that
>>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>>> have
>>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>>> immediately
>>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> http://hortonworks.com/download/
>>>>>>>> 
>>>>>>>> --
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity
>>>>>>> to
>>>>>>>> which it is addressed and may contain information that is
>>>>> confidential,
>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>>> that
>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>>> received this communication in error, please contact the sender
>>>>>>> immediately
>>>>>>>> and delete it from your system. Thank You.
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>> 
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>>> of this message is not the intended recipient, you are hereby notified that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender immediately
>>>>> and delete it from your system. Thank You.
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/hdp/
>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

>From YARN, we have a couple of items related to log-aggregation of services
and node-labels that are not tracked as blockers, but are working on
targeting 2.6. I am trying to get them in by end of this week.

+Vinod

On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> 2.6.0 is close now.
>
> Here are the remaining blockers, I'm hoping cut an RC in the next week or
> so:
> http://s.apache.org/hadoop-2.6.0-blockers
>
> thanks,
> Arun
>
> On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > Folks,
> >
> >  I've created branch-2.6 to stabilize the release.
> >
> >  Committers, please exercise caution henceforth on commits other than
> the ones we've discussed on this thread already.
> >
> >  By default new features should now be targeted to the version "2.7"
> henceforth - I've ensure all the projects have that version on jira.
> >
> > thanks,
> > Arun
> >
> > On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> >
> >> Sounds good. I'll branch this weekend and we can merge the jiras we
> >> discussed in this thread as they they get wrapped next week.
> >>
> >> Thanks everyone.
> >>
> >> Arun
> >>
> >>
> >>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >>>
> >>> We can branch off in a week or two so that work on branch-2 itself can
> go
> >>> ahead with other features that can't fit in 2.6. Independent of that,
> we
> >>> can then decide on the timeline of the release candidates once
> branch-2.6
> >>> is close to being done w.r.t the planned features.
> >>>
> >>> Branching it off can let us focus on specific features that we want in
> for
> >>> 2.6 and then eventually blockers for the release, nothing else. There
> is a
> >>> trivial pain of committing to one more branch, but it's worth it in
> this
> >>> case IMO.
> >>>
> >>> A lot of efforts are happening in parallel from the YARN side from
> where I
> >>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if
> we
> >>> don't branch off and selectively try to get stuff in, it is likely to
> be in
> >>> a perpetual delay.
> >>>
> >>> My 2 cents.
> >>>
> >>> +Vinod
> >>>
> >>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <
> suresh@hortonworks.com>
> >>> wrote:
> >>>
> >>>> Given some of the features are in final stages of stabilization,
> >>>> Arun, we should hold off creating 2.6 branch or building an RC by a
> week?
> >>>> All the features in flux are important ones and worth delaying the
> release
> >>>> by a week.
> >>>>
> >>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> Hey Nicholas,
> >>>>>
> >>>>> My concern about Archival Storage isn't related to the code quality
> or
> >>>> the
> >>>>> size of the feature. I think that you and Jing did good work. My
> concern
> >>>> is
> >>>>> that once we ship, we're locked into that set of archival storage
> APIs,
> >>>> and
> >>>>> these APIs are not yet finalized. Simply being able to turn off the
> >>>> feature
> >>>>> does not change the compatibility story.
> >>>>>
> >>>>> I'm willing to devote time to help review these JIRAs and kick the
> tires
> >>>> on
> >>>>> the APIs, but my point above was that I'm not sure it'd all be done
> by
> >>>> the
> >>>>> end of the week. Testing might also reveal additional changes that
> need
> >>>> to
> >>>>> be made, which also might not happen by end-of-week.
> >>>>>
> >>>>> I guess the question before us is if we're comfortable putting
> something
> >>>> in
> >>>>> branch-2.6 and then potentially adding API changes after. I'm okay
> with
> >>>>> that as long as we're all aware that this might happen.
> >>>>>
> >>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
> >>>> fine
> >>>>> with it's inclusion, just a heads up that we might need some extra
> time
> >>>> to
> >>>>> finalize things before an RC can be cut.
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>>
> >>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> >>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am worry about KMS and transparent encryption since there are
> quite
> >>>>> many
> >>>>>> bugs discovered after it got merged to branch-2.  It gives us an
> >>>>> impression
> >>>>>> that the feature is not yet well tested.  Indeed, transparent
> >>>> encryption
> >>>>> is
> >>>>>> a complicated feature which changes the core part of HDFS.  It is
> not
> >>>>> easy
> >>>>>> to get everything right.
> >>>>>>
> >>>>>>
> >>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low
> risk
> >>>>>> feature.  It introduces a new storage type ARCHIVE and the concept
> of
> >>>>> block
> >>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
> >>>>> storage,
> >>>>>> the blocks will be stored using the appropriate storage types
> specified
> >>>>> by
> >>>>>> storage policies assigned to the files/directories.  Cluster admin
> >>>> could
> >>>>>> disable the feature by simply not configuring any storage type and
> not
> >>>>>> setting any storage policy as before.   As Suresh mentioned,
> HDFS-6584
> >>>> is
> >>>>>> in the final stages to be merged to branch-2.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Tsz-Wo
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> >>>>>> suresh@hortonworks.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I actually would like to see both archival storage and single
> replica
> >>>>>>> memory writes to be in 2.6 release. Archival storage is in the
> final
> >>>>>> stages
> >>>>>>> of getting ready for branch-2 merge as Nicholas has already
> indicated
> >>>> on
> >>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both
> of
> >>>>> these
> >>>>>>> features are being in development for sometime.
> >>>>>>>
> >>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> >>>> andrew.wang@cloudera.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hey Arun,
> >>>>>>>>
> >>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
> >>>>>> add/retarget
> >>>>>>>> things accordingly?
> >>>>>>>>
> >>>>>>>> I think the KMS and transparent encryption are ready to go. We've
> >>>> got
> >>>>> a
> >>>>>>>> very few further bug fixes pending, but that's it.
> >>>>>>>>
> >>>>>>>> Two HDFS things that I think probably won't make the end of the
> week
> >>>>> are
> >>>>>>>> archival storage (HDFS-6584) and single replica memory writes
> >>>>>> (HDFS-6581),
> >>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just
> merged
> >>>> to
> >>>>>>>> trunk and I think needs a little more work before it goes into
> >>>>> branch-2.
> >>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> >>>> further
> >>>>>> off
> >>>>>>>> yet.
> >>>>>>>>
> >>>>>>>> Just my 2c as I did not work directly on these features. I just
> >>>>>> generally
> >>>>>>>> shy away from shipping bits quite this fresh.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <acm@hortonworks.com
> >
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
> >>>>>> nicely.
> >>>>>>>>>
> >>>>>>>>> I'll create branch-2.6 by end of the week and we can go from
> there
> >>>>> to
> >>>>>>>>> stabilize it - hopefully in the next few weeks.
> >>>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> >>>> acm@hortonworks.com
> >>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Folks,
> >>>>>>>>>>
> >>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
> >>>> to
> >>>>>>>>>> hadoop-2.6.
> >>>>>>>>>>
> >>>>>>>>>> Currently, here is the Roadmap per the wiki:
> >>>>>>>>>>
> >>>>>>>>>>       • HADOOP
> >>>>>>>>>>               • Credential provider HADOOP-10607
> >>>>>>>>>>       • HDFS
> >>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
> >>>>> for
> >>>>>>>>> using
> >>>>>>>>>> storage tiers by the applications HDFS-5682
> >>>>>>>>>>               • Memory as storage tier HDFS-5851
> >>>>>>>>>>       • YARN
> >>>>>>>>>>               • Dynamic Resource Configuration YARN-291
> >>>>>>>>>>               • NodeManager Restart YARN-1336
> >>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
> >>>>>>>>>>               • Support for admin-specified labels in YARN
> >>>>>> YARN-796
> >>>>>>>>>>               • Support for automatic, shared cache for YARN
> >>>>>>>>> application
> >>>>>>>>>> artifacts YARN-1492
> >>>>>>>>>>               • Support NodeGroup layer topology on YARN
> >>>> YARN-18
> >>>>>>>>>>               • Support for Docker containers in YARN
> >>>> YARN-1964
> >>>>>>>>>>               • YARN service registry YARN-913
> >>>>>>>>>>
> >>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
> >>>>>> won't.
> >>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
> >>>>>> would
> >>>>>>>> be
> >>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> >>>> up a
> >>>>>>>>> cadence.
> >>>>>>>>>>
> >>>>>>>>>> More importantly, as we discussed previously, we'd like
> >>>>> hadoop-2.6
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
> >>>>> start
> >>>>>> a
> >>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> >>>>> and
> >>>>>>>> see
> >>>>>>>>>> how they feel about this.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>> entity
> >>>>>>>> to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> the
> >>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> http://hortonworks.com/download/
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>>> to
> >>>>>>> which it is addressed and may contain information that is
> >>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://hortonworks.com/download/
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>>> of this message is not the intended recipient, you are hereby
> notified that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> immediately
> >>>> and delete it from your system. Thank You.
> >>>
> >>> --
> >>> CONFIDENTIALITY NOTICE
> >>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>> which it is addressed and may contain information that is confidential,
> >>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>> of this message is not the intended recipient, you are hereby notified
> that
> >>> any printing, copying, dissemination, distribution, disclosure or
> >>> forwarding of this communication is strictly prohibited. If you have
> >>> received this communication in error, please contact the sender
> immediately
> >>> and delete it from your system. Thank You.
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/hdp/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

2.6.0 is close now.

Here are the remaining blockers, I'm hoping cut an RC in the next week or so:
http://s.apache.org/hadoop-2.6.0-blockers

thanks,
Arun

On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
> 
>  I've created branch-2.6 to stabilize the release.
>  
>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
> 
>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
> 
> thanks,
> Arun
> 
> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> 
>> Sounds good. I'll branch this weekend and we can merge the jiras we
>> discussed in this thread as they they get wrapped next week.
>> 
>> Thanks everyone.
>> 
>> Arun
>> 
>> 
>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> We can branch off in a week or two so that work on branch-2 itself can go
>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>> can then decide on the timeline of the release candidates once branch-2.6
>>> is close to being done w.r.t the planned features.
>>> 
>>> Branching it off can let us focus on specific features that we want in for
>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>> trivial pain of committing to one more branch, but it's worth it in this
>>> case IMO.
>>> 
>>> A lot of efforts are happening in parallel from the YARN side from where I
>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>> a perpetual delay.
>>> 
>>> My 2 cents.
>>> 
>>> +Vinod
>>> 
>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>> wrote:
>>> 
>>>> Given some of the features are in final stages of stabilization,
>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>> All the features in flux are important ones and worth delaying the release
>>>> by a week.
>>>> 
>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> Hey Nicholas,
>>>>> 
>>>>> My concern about Archival Storage isn't related to the code quality or
>>>> the
>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>> is
>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>> and
>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>> feature
>>>>> does not change the compatibility story.
>>>>> 
>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>> on
>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>> the
>>>>> end of the week. Testing might also reveal additional changes that need
>>>> to
>>>>> be made, which also might not happen by end-of-week.
>>>>> 
>>>>> I guess the question before us is if we're comfortable putting something
>>>> in
>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>> that as long as we're all aware that this might happen.
>>>>> 
>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>> fine
>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>> to
>>>>> finalize things before an RC can be cut.
>>>>> 
>>>>> Thanks,
>>>>> Andrew
>>>>> 
>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>> many
>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>> impression
>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>> encryption
>>>>> is
>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>> easy
>>>>>> to get everything right.
>>>>>> 
>>>>>> 
>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>> block
>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>> storage,
>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>> by
>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>> could
>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>> is
>>>>>> in the final stages to be merged to branch-2.
>>>>>> 
>>>>>> Regards,
>>>>>> Tsz-Wo
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>> suresh@hortonworks.com> wrote:
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>> stages
>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>> on
>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>> these
>>>>>>> features are being in development for sometime.
>>>>>>> 
>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>> andrew.wang@cloudera.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey Arun,
>>>>>>>> 
>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>> add/retarget
>>>>>>>> things accordingly?
>>>>>>>> 
>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>> got
>>>>> a
>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>> 
>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>> are
>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>> (HDFS-6581),
>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>> to
>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>> branch-2.
>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>> further
>>>>>> off
>>>>>>>> yet.
>>>>>>>> 
>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>> generally
>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>> nicely.
>>>>>>>>> 
>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>> to
>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>> acm@hortonworks.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Folks,
>>>>>>>>>> 
>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>> to
>>>>>>>>>> hadoop-2.6.
>>>>>>>>>> 
>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>> 
>>>>>>>>>>       • HADOOP
>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>       • HDFS
>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>> for
>>>>>>>>> using
>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>       • YARN
>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>> YARN-796
>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>> application
>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>> YARN-18
>>>>>>>>>>               • Support for Docker containers in YARN
>>>> YARN-1964
>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>> 
>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>> won't.
>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>> would
>>>>>>>> be
>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>> up a
>>>>>>>>> cadence.
>>>>>>>>>> 
>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>> hadoop-2.6
>>>>>> to
>>>>>>>> be
>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>> start
>>>>>> a
>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>> and
>>>>>>>> see
>>>>>>>>>> how they feel about this.
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity
>>>>>>>> to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>> 
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> http://hortonworks.com/download/
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

Re-sending… doesn't look like it made through.

On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
> 
>  I've created branch-2.6 to stabilize the release.
>  
>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
> 
>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
> 
> thanks,
> Arun
> 
> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> 
>> Sounds good. I'll branch this weekend and we can merge the jiras we
>> discussed in this thread as they they get wrapped next week.
>> 
>> Thanks everyone.
>> 
>> Arun
>> 
>> 
>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> We can branch off in a week or two so that work on branch-2 itself can go
>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>> can then decide on the timeline of the release candidates once branch-2.6
>>> is close to being done w.r.t the planned features.
>>> 
>>> Branching it off can let us focus on specific features that we want in for
>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>> trivial pain of committing to one more branch, but it's worth it in this
>>> case IMO.
>>> 
>>> A lot of efforts are happening in parallel from the YARN side from where I
>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>> a perpetual delay.
>>> 
>>> My 2 cents.
>>> 
>>> +Vinod
>>> 
>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>> wrote:
>>> 
>>>> Given some of the features are in final stages of stabilization,
>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>> All the features in flux are important ones and worth delaying the release
>>>> by a week.
>>>> 
>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> Hey Nicholas,
>>>>> 
>>>>> My concern about Archival Storage isn't related to the code quality or
>>>> the
>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>> is
>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>> and
>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>> feature
>>>>> does not change the compatibility story.
>>>>> 
>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>> on
>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>> the
>>>>> end of the week. Testing might also reveal additional changes that need
>>>> to
>>>>> be made, which also might not happen by end-of-week.
>>>>> 
>>>>> I guess the question before us is if we're comfortable putting something
>>>> in
>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>> that as long as we're all aware that this might happen.
>>>>> 
>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>> fine
>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>> to
>>>>> finalize things before an RC can be cut.
>>>>> 
>>>>> Thanks,
>>>>> Andrew
>>>>> 
>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>> many
>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>> impression
>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>> encryption
>>>>> is
>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>> easy
>>>>>> to get everything right.
>>>>>> 
>>>>>> 
>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>> block
>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>> storage,
>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>> by
>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>> could
>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>> is
>>>>>> in the final stages to be merged to branch-2.
>>>>>> 
>>>>>> Regards,
>>>>>> Tsz-Wo
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>> suresh@hortonworks.com> wrote:
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>> stages
>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>> on
>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>> these
>>>>>>> features are being in development for sometime.
>>>>>>> 
>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>> andrew.wang@cloudera.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey Arun,
>>>>>>>> 
>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>> add/retarget
>>>>>>>> things accordingly?
>>>>>>>> 
>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>> got
>>>>> a
>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>> 
>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>> are
>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>> (HDFS-6581),
>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>> to
>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>> branch-2.
>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>> further
>>>>>> off
>>>>>>>> yet.
>>>>>>>> 
>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>> generally
>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>> nicely.
>>>>>>>>> 
>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>> to
>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>> acm@hortonworks.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Folks,
>>>>>>>>>> 
>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>> to
>>>>>>>>>> hadoop-2.6.
>>>>>>>>>> 
>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>> 
>>>>>>>>>>       • HADOOP
>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>       • HDFS
>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>> for
>>>>>>>>> using
>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>       • YARN
>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>> YARN-796
>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>> application
>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>> YARN-18
>>>>>>>>>>               • Support for Docker containers in YARN
>>>> YARN-1964
>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>> 
>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>> won't.
>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>> would
>>>>>>>> be
>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>> up a
>>>>>>>>> cadence.
>>>>>>>>>> 
>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>> hadoop-2.6
>>>>>> to
>>>>>>>> be
>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>> start
>>>>>> a
>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>> and
>>>>>>>> see
>>>>>>>>>> how they feel about this.
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity
>>>>>>>> to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>> 
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> http://hortonworks.com/download/
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

2.6.0 is close now.

Here are the remaining blockers, I'm hoping cut an RC in the next week or so:
http://s.apache.org/hadoop-2.6.0-blockers

thanks,
Arun

On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
> 
>  I've created branch-2.6 to stabilize the release.
>  
>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
> 
>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
> 
> thanks,
> Arun
> 
> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> 
>> Sounds good. I'll branch this weekend and we can merge the jiras we
>> discussed in this thread as they they get wrapped next week.
>> 
>> Thanks everyone.
>> 
>> Arun
>> 
>> 
>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> We can branch off in a week or two so that work on branch-2 itself can go
>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>> can then decide on the timeline of the release candidates once branch-2.6
>>> is close to being done w.r.t the planned features.
>>> 
>>> Branching it off can let us focus on specific features that we want in for
>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>> trivial pain of committing to one more branch, but it's worth it in this
>>> case IMO.
>>> 
>>> A lot of efforts are happening in parallel from the YARN side from where I
>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>> a perpetual delay.
>>> 
>>> My 2 cents.
>>> 
>>> +Vinod
>>> 
>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>> wrote:
>>> 
>>>> Given some of the features are in final stages of stabilization,
>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>> All the features in flux are important ones and worth delaying the release
>>>> by a week.
>>>> 
>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> Hey Nicholas,
>>>>> 
>>>>> My concern about Archival Storage isn't related to the code quality or
>>>> the
>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>> is
>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>> and
>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>> feature
>>>>> does not change the compatibility story.
>>>>> 
>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>> on
>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>> the
>>>>> end of the week. Testing might also reveal additional changes that need
>>>> to
>>>>> be made, which also might not happen by end-of-week.
>>>>> 
>>>>> I guess the question before us is if we're comfortable putting something
>>>> in
>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>> that as long as we're all aware that this might happen.
>>>>> 
>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>> fine
>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>> to
>>>>> finalize things before an RC can be cut.
>>>>> 
>>>>> Thanks,
>>>>> Andrew
>>>>> 
>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>> many
>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>> impression
>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>> encryption
>>>>> is
>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>> easy
>>>>>> to get everything right.
>>>>>> 
>>>>>> 
>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>> block
>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>> storage,
>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>> by
>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>> could
>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>> is
>>>>>> in the final stages to be merged to branch-2.
>>>>>> 
>>>>>> Regards,
>>>>>> Tsz-Wo
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>> suresh@hortonworks.com> wrote:
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>> stages
>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>> on
>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>> these
>>>>>>> features are being in development for sometime.
>>>>>>> 
>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>> andrew.wang@cloudera.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey Arun,
>>>>>>>> 
>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>> add/retarget
>>>>>>>> things accordingly?
>>>>>>>> 
>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>> got
>>>>> a
>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>> 
>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>> are
>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>> (HDFS-6581),
>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>> to
>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>> branch-2.
>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>> further
>>>>>> off
>>>>>>>> yet.
>>>>>>>> 
>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>> generally
>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>> nicely.
>>>>>>>>> 
>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>> to
>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>> acm@hortonworks.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Folks,
>>>>>>>>>> 
>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>> to
>>>>>>>>>> hadoop-2.6.
>>>>>>>>>> 
>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>> 
>>>>>>>>>>       • HADOOP
>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>       • HDFS
>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>> for
>>>>>>>>> using
>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>       • YARN
>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>> YARN-796
>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>> application
>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>> YARN-18
>>>>>>>>>>               • Support for Docker containers in YARN
>>>> YARN-1964
>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>> 
>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>> won't.
>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>> would
>>>>>>>> be
>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>> up a
>>>>>>>>> cadence.
>>>>>>>>>> 
>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>> hadoop-2.6
>>>>>> to
>>>>>>>> be
>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>> start
>>>>>> a
>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>> and
>>>>>>>> see
>>>>>>>>>> how they feel about this.
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity
>>>>>>>> to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>> 
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> http://hortonworks.com/download/
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

Re-sending… doesn't look like it made through.

On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
> 
>  I've created branch-2.6 to stabilize the release.
>  
>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
> 
>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
> 
> thanks,
> Arun
> 
> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> 
>> Sounds good. I'll branch this weekend and we can merge the jiras we
>> discussed in this thread as they they get wrapped next week.
>> 
>> Thanks everyone.
>> 
>> Arun
>> 
>> 
>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> We can branch off in a week or two so that work on branch-2 itself can go
>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>> can then decide on the timeline of the release candidates once branch-2.6
>>> is close to being done w.r.t the planned features.
>>> 
>>> Branching it off can let us focus on specific features that we want in for
>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>> trivial pain of committing to one more branch, but it's worth it in this
>>> case IMO.
>>> 
>>> A lot of efforts are happening in parallel from the YARN side from where I
>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>> a perpetual delay.
>>> 
>>> My 2 cents.
>>> 
>>> +Vinod
>>> 
>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>> wrote:
>>> 
>>>> Given some of the features are in final stages of stabilization,
>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>> All the features in flux are important ones and worth delaying the release
>>>> by a week.
>>>> 
>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> Hey Nicholas,
>>>>> 
>>>>> My concern about Archival Storage isn't related to the code quality or
>>>> the
>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>> is
>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>> and
>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>> feature
>>>>> does not change the compatibility story.
>>>>> 
>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>> on
>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>> the
>>>>> end of the week. Testing might also reveal additional changes that need
>>>> to
>>>>> be made, which also might not happen by end-of-week.
>>>>> 
>>>>> I guess the question before us is if we're comfortable putting something
>>>> in
>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>> that as long as we're all aware that this might happen.
>>>>> 
>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>> fine
>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>> to
>>>>> finalize things before an RC can be cut.
>>>>> 
>>>>> Thanks,
>>>>> Andrew
>>>>> 
>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>> many
>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>> impression
>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>> encryption
>>>>> is
>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>> easy
>>>>>> to get everything right.
>>>>>> 
>>>>>> 
>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>> block
>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>> storage,
>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>> by
>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>> could
>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>> is
>>>>>> in the final stages to be merged to branch-2.
>>>>>> 
>>>>>> Regards,
>>>>>> Tsz-Wo
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>> suresh@hortonworks.com> wrote:
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>> stages
>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>> on
>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>> these
>>>>>>> features are being in development for sometime.
>>>>>>> 
>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>> andrew.wang@cloudera.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey Arun,
>>>>>>>> 
>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>> add/retarget
>>>>>>>> things accordingly?
>>>>>>>> 
>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>> got
>>>>> a
>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>> 
>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>> are
>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>> (HDFS-6581),
>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>> to
>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>> branch-2.
>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>> further
>>>>>> off
>>>>>>>> yet.
>>>>>>>> 
>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>> generally
>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>> nicely.
>>>>>>>>> 
>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>> to
>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>> acm@hortonworks.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Folks,
>>>>>>>>>> 
>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>> to
>>>>>>>>>> hadoop-2.6.
>>>>>>>>>> 
>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>> 
>>>>>>>>>>       • HADOOP
>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>       • HDFS
>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>> for
>>>>>>>>> using
>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>       • YARN
>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>> YARN-796
>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>> application
>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>> YARN-18
>>>>>>>>>>               • Support for Docker containers in YARN
>>>> YARN-1964
>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>> 
>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>> won't.
>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>> would
>>>>>>>> be
>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>> up a
>>>>>>>>> cadence.
>>>>>>>>>> 
>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>> hadoop-2.6
>>>>>> to
>>>>>>>> be
>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>> start
>>>>>> a
>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>> and
>>>>>>>> see
>>>>>>>>>> how they feel about this.
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity
>>>>>>>> to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>> 
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> http://hortonworks.com/download/
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

2.6.0 is close now.

Here are the remaining blockers, I'm hoping cut an RC in the next week or so:
http://s.apache.org/hadoop-2.6.0-blockers

thanks,
Arun

On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
> 
>  I've created branch-2.6 to stabilize the release.
>  
>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
> 
>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
> 
> thanks,
> Arun
> 
> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> 
>> Sounds good. I'll branch this weekend and we can merge the jiras we
>> discussed in this thread as they they get wrapped next week.
>> 
>> Thanks everyone.
>> 
>> Arun
>> 
>> 
>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> We can branch off in a week or two so that work on branch-2 itself can go
>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>> can then decide on the timeline of the release candidates once branch-2.6
>>> is close to being done w.r.t the planned features.
>>> 
>>> Branching it off can let us focus on specific features that we want in for
>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>> trivial pain of committing to one more branch, but it's worth it in this
>>> case IMO.
>>> 
>>> A lot of efforts are happening in parallel from the YARN side from where I
>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>> a perpetual delay.
>>> 
>>> My 2 cents.
>>> 
>>> +Vinod
>>> 
>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>> wrote:
>>> 
>>>> Given some of the features are in final stages of stabilization,
>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>> All the features in flux are important ones and worth delaying the release
>>>> by a week.
>>>> 
>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> Hey Nicholas,
>>>>> 
>>>>> My concern about Archival Storage isn't related to the code quality or
>>>> the
>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>> is
>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>> and
>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>> feature
>>>>> does not change the compatibility story.
>>>>> 
>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>> on
>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>> the
>>>>> end of the week. Testing might also reveal additional changes that need
>>>> to
>>>>> be made, which also might not happen by end-of-week.
>>>>> 
>>>>> I guess the question before us is if we're comfortable putting something
>>>> in
>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>> that as long as we're all aware that this might happen.
>>>>> 
>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>> fine
>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>> to
>>>>> finalize things before an RC can be cut.
>>>>> 
>>>>> Thanks,
>>>>> Andrew
>>>>> 
>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>> many
>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>> impression
>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>> encryption
>>>>> is
>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>> easy
>>>>>> to get everything right.
>>>>>> 
>>>>>> 
>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>> block
>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>> storage,
>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>> by
>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>> could
>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>> is
>>>>>> in the final stages to be merged to branch-2.
>>>>>> 
>>>>>> Regards,
>>>>>> Tsz-Wo
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>> suresh@hortonworks.com> wrote:
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>> stages
>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>> on
>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>> these
>>>>>>> features are being in development for sometime.
>>>>>>> 
>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>> andrew.wang@cloudera.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey Arun,
>>>>>>>> 
>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>> add/retarget
>>>>>>>> things accordingly?
>>>>>>>> 
>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>> got
>>>>> a
>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>> 
>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>> are
>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>> (HDFS-6581),
>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>> to
>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>> branch-2.
>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>> further
>>>>>> off
>>>>>>>> yet.
>>>>>>>> 
>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>> generally
>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>> nicely.
>>>>>>>>> 
>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>> to
>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>> acm@hortonworks.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Folks,
>>>>>>>>>> 
>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>> to
>>>>>>>>>> hadoop-2.6.
>>>>>>>>>> 
>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>> 
>>>>>>>>>>       • HADOOP
>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>       • HDFS
>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>> for
>>>>>>>>> using
>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>       • YARN
>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>> YARN-796
>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>> application
>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>> YARN-18
>>>>>>>>>>               • Support for Docker containers in YARN
>>>> YARN-1964
>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>> 
>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>> won't.
>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>> would
>>>>>>>> be
>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>> up a
>>>>>>>>> cadence.
>>>>>>>>>> 
>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>> hadoop-2.6
>>>>>> to
>>>>>>>> be
>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>> start
>>>>>> a
>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>> and
>>>>>>>> see
>>>>>>>>>> how they feel about this.
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity
>>>>>>>> to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>> 
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> http://hortonworks.com/download/
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

Re-sending… doesn't look like it made through.

On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
> 
>  I've created branch-2.6 to stabilize the release.
>  
>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
> 
>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
> 
> thanks,
> Arun
> 
> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> 
>> Sounds good. I'll branch this weekend and we can merge the jiras we
>> discussed in this thread as they they get wrapped next week.
>> 
>> Thanks everyone.
>> 
>> Arun
>> 
>> 
>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> We can branch off in a week or two so that work on branch-2 itself can go
>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>> can then decide on the timeline of the release candidates once branch-2.6
>>> is close to being done w.r.t the planned features.
>>> 
>>> Branching it off can let us focus on specific features that we want in for
>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>> trivial pain of committing to one more branch, but it's worth it in this
>>> case IMO.
>>> 
>>> A lot of efforts are happening in parallel from the YARN side from where I
>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>> a perpetual delay.
>>> 
>>> My 2 cents.
>>> 
>>> +Vinod
>>> 
>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>> wrote:
>>> 
>>>> Given some of the features are in final stages of stabilization,
>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>> All the features in flux are important ones and worth delaying the release
>>>> by a week.
>>>> 
>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> Hey Nicholas,
>>>>> 
>>>>> My concern about Archival Storage isn't related to the code quality or
>>>> the
>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>> is
>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>> and
>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>> feature
>>>>> does not change the compatibility story.
>>>>> 
>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>> on
>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>> the
>>>>> end of the week. Testing might also reveal additional changes that need
>>>> to
>>>>> be made, which also might not happen by end-of-week.
>>>>> 
>>>>> I guess the question before us is if we're comfortable putting something
>>>> in
>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>> that as long as we're all aware that this might happen.
>>>>> 
>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>> fine
>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>> to
>>>>> finalize things before an RC can be cut.
>>>>> 
>>>>> Thanks,
>>>>> Andrew
>>>>> 
>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>> many
>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>> impression
>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>> encryption
>>>>> is
>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>> easy
>>>>>> to get everything right.
>>>>>> 
>>>>>> 
>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>> block
>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>> storage,
>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>> by
>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>> could
>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>> is
>>>>>> in the final stages to be merged to branch-2.
>>>>>> 
>>>>>> Regards,
>>>>>> Tsz-Wo
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>> suresh@hortonworks.com> wrote:
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>> stages
>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>> on
>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>> these
>>>>>>> features are being in development for sometime.
>>>>>>> 
>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>> andrew.wang@cloudera.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey Arun,
>>>>>>>> 
>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>> add/retarget
>>>>>>>> things accordingly?
>>>>>>>> 
>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>> got
>>>>> a
>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>> 
>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>> are
>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>> (HDFS-6581),
>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>> to
>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>> branch-2.
>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>> further
>>>>>> off
>>>>>>>> yet.
>>>>>>>> 
>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>> generally
>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>> nicely.
>>>>>>>>> 
>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>> to
>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>> acm@hortonworks.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Folks,
>>>>>>>>>> 
>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>> to
>>>>>>>>>> hadoop-2.6.
>>>>>>>>>> 
>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>> 
>>>>>>>>>>       • HADOOP
>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>       • HDFS
>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>> for
>>>>>>>>> using
>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>       • YARN
>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>> YARN-796
>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>> application
>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>> YARN-18
>>>>>>>>>>               • Support for Docker containers in YARN
>>>> YARN-1964
>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>> 
>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>> won't.
>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>> would
>>>>>>>> be
>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>> up a
>>>>>>>>> cadence.
>>>>>>>>>> 
>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>> hadoop-2.6
>>>>>> to
>>>>>>>> be
>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>> start
>>>>>> a
>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>> and
>>>>>>>> see
>>>>>>>>>> how they feel about this.
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity
>>>>>>>> to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>> 
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> http://hortonworks.com/download/
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun C Murthy <ac...@hortonworks.com>.

2.6.0 is close now.

Here are the remaining blockers, I'm hoping cut an RC in the next week or so:
http://s.apache.org/hadoop-2.6.0-blockers

thanks,
Arun

On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
> 
>  I've created branch-2.6 to stabilize the release.
>  
>  Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
> 
>  By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
> 
> thanks,
> Arun
> 
> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> 
>> Sounds good. I'll branch this weekend and we can merge the jiras we
>> discussed in this thread as they they get wrapped next week.
>> 
>> Thanks everyone.
>> 
>> Arun
>> 
>> 
>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> We can branch off in a week or two so that work on branch-2 itself can go
>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>> can then decide on the timeline of the release candidates once branch-2.6
>>> is close to being done w.r.t the planned features.
>>> 
>>> Branching it off can let us focus on specific features that we want in for
>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>> trivial pain of committing to one more branch, but it's worth it in this
>>> case IMO.
>>> 
>>> A lot of efforts are happening in parallel from the YARN side from where I
>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>> a perpetual delay.
>>> 
>>> My 2 cents.
>>> 
>>> +Vinod
>>> 
>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>> wrote:
>>> 
>>>> Given some of the features are in final stages of stabilization,
>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>> All the features in flux are important ones and worth delaying the release
>>>> by a week.
>>>> 
>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> Hey Nicholas,
>>>>> 
>>>>> My concern about Archival Storage isn't related to the code quality or
>>>> the
>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>> is
>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>> and
>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>> feature
>>>>> does not change the compatibility story.
>>>>> 
>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>> on
>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>> the
>>>>> end of the week. Testing might also reveal additional changes that need
>>>> to
>>>>> be made, which also might not happen by end-of-week.
>>>>> 
>>>>> I guess the question before us is if we're comfortable putting something
>>>> in
>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>> that as long as we're all aware that this might happen.
>>>>> 
>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>> fine
>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>> to
>>>>> finalize things before an RC can be cut.
>>>>> 
>>>>> Thanks,
>>>>> Andrew
>>>>> 
>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>> many
>>>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>>>> impression
>>>>>> that the feature is not yet well tested.  Indeed, transparent
>>>> encryption
>>>>> is
>>>>>> a complicated feature which changes the core part of HDFS.  It is not
>>>>> easy
>>>>>> to get everything right.
>>>>>> 
>>>>>> 
>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>>>> block
>>>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>>>> storage,
>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>> by
>>>>>> storage policies assigned to the files/directories.  Cluster admin
>>>> could
>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>>>> is
>>>>>> in the final stages to be merged to branch-2.
>>>>>> 
>>>>>> Regards,
>>>>>> Tsz-Wo
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>> suresh@hortonworks.com> wrote:
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>> stages
>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>> on
>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>> these
>>>>>>> features are being in development for sometime.
>>>>>>> 
>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>> andrew.wang@cloudera.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey Arun,
>>>>>>>> 
>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>> add/retarget
>>>>>>>> things accordingly?
>>>>>>>> 
>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>> got
>>>>> a
>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>> 
>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>> are
>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>> (HDFS-6581),
>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>> to
>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>> branch-2.
>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>> further
>>>>>> off
>>>>>>>> yet.
>>>>>>>> 
>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>> generally
>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>> 
>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>> nicely.
>>>>>>>>> 
>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>> to
>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>> acm@hortonworks.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Folks,
>>>>>>>>>> 
>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>> to
>>>>>>>>>> hadoop-2.6.
>>>>>>>>>> 
>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>> 
>>>>>>>>>>       • HADOOP
>>>>>>>>>>               • Credential provider HADOOP-10607
>>>>>>>>>>       • HDFS
>>>>>>>>>>               • Heterogeneous storage (Phase 2) - Support APIs
>>>>> for
>>>>>>>>> using
>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>>               • Memory as storage tier HDFS-5851
>>>>>>>>>>       • YARN
>>>>>>>>>>               • Dynamic Resource Configuration YARN-291
>>>>>>>>>>               • NodeManager Restart YARN-1336
>>>>>>>>>>               • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>>               • Support for admin-specified labels in YARN
>>>>>> YARN-796
>>>>>>>>>>               • Support for automatic, shared cache for YARN
>>>>>>>>> application
>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>>               • Support NodeGroup layer topology on YARN
>>>> YARN-18
>>>>>>>>>>               • Support for Docker containers in YARN
>>>> YARN-1964
>>>>>>>>>>               • YARN service registry YARN-913
>>>>>>>>>> 
>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>> won't.
>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>> would
>>>>>>>> be
>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>> up a
>>>>>>>>> cadence.
>>>>>>>>>> 
>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>> hadoop-2.6
>>>>>> to
>>>>>>>> be
>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>> start
>>>>>> a
>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>> and
>>>>>>>> see
>>>>>>>>>> how they feel about this.
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity
>>>>>>>> to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>> 
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> http://hortonworks.com/download/
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun Murthy <ac...@hortonworks.com>.

Sounds good. I'll branch this weekend and we can merge the jiras we
discussed in this thread as they they get wrapped next week.

Thanks everyone.

Arun


> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> We can branch off in a week or two so that work on branch-2 itself can go
> ahead with other features that can't fit in 2.6. Independent of that, we
> can then decide on the timeline of the release candidates once branch-2.6
> is close to being done w.r.t the planned features.
>
> Branching it off can let us focus on specific features that we want in for
> 2.6 and then eventually blockers for the release, nothing else. There is a
> trivial pain of committing to one more branch, but it's worth it in this
> case IMO.
>
> A lot of efforts are happening in parallel from the YARN side from where I
> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
> don't branch off and selectively try to get stuff in, it is likely to be in
> a perpetual delay.
>
> My 2 cents.
>
> +Vinod
>
> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
>> Given some of the features are in final stages of stabilization,
>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>> All the features in flux are important ones and worth delaying the release
>> by a week.
>>
>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>>
>>> Hey Nicholas,
>>>
>>> My concern about Archival Storage isn't related to the code quality or
>> the
>>> size of the feature. I think that you and Jing did good work. My concern
>> is
>>> that once we ship, we're locked into that set of archival storage APIs,
>> and
>>> these APIs are not yet finalized. Simply being able to turn off the
>> feature
>>> does not change the compatibility story.
>>>
>>> I'm willing to devote time to help review these JIRAs and kick the tires
>> on
>>> the APIs, but my point above was that I'm not sure it'd all be done by
>> the
>>> end of the week. Testing might also reveal additional changes that need
>> to
>>> be made, which also might not happen by end-of-week.
>>>
>>> I guess the question before us is if we're comfortable putting something
>> in
>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>> that as long as we're all aware that this might happen.
>>>
>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>> fine
>>> with it's inclusion, just a heads up that we might need some extra time
>> to
>>> finalize things before an RC can be cut.
>>>
>>> Thanks,
>>> Andrew
>>>
>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am worry about KMS and transparent encryption since there are quite
>>> many
>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>> impression
>>>> that the feature is not yet well tested.  Indeed, transparent
>> encryption
>>> is
>>>> a complicated feature which changes the core part of HDFS.  It is not
>>> easy
>>>> to get everything right.
>>>>
>>>>
>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>> block
>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>> storage,
>>>> the blocks will be stored using the appropriate storage types specified
>>> by
>>>> storage policies assigned to the files/directories.  Cluster admin
>> could
>>>> disable the feature by simply not configuring any storage type and not
>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>> is
>>>> in the final stages to be merged to branch-2.
>>>>
>>>> Regards,
>>>> Tsz-Wo
>>>>
>>>>
>>>>
>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>> suresh@hortonworks.com> wrote:
>>>>
>>>>
>>>>>
>>>>>
>>>>> I actually would like to see both archival storage and single replica
>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>> stages
>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>> on
>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>> these
>>>>> features are being in development for sometime.
>>>>>
>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>> andrew.wang@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Hey Arun,
>>>>>>
>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>> add/retarget
>>>>>> things accordingly?
>>>>>>
>>>>>> I think the KMS and transparent encryption are ready to go. We've
>> got
>>> a
>>>>>> very few further bug fixes pending, but that's it.
>>>>>>
>>>>>> Two HDFS things that I think probably won't make the end of the week
>>> are
>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>> (HDFS-6581),
>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>> to
>>>>>> trunk and I think needs a little more work before it goes into
>>> branch-2.
>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>> further
>>>> off
>>>>>> yet.
>>>>>>
>>>>>> Just my 2c as I did not work directly on these features. I just
>>>> generally
>>>>>> shy away from shipping bits quite this fresh.
>>>>>>
>>>>>> Thanks,
>>>>>> Andrew
>>>>>>
>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>> wrote:
>>>>>>
>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>> nicely.
>>>>>>>
>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>> to
>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> thanks,
>>>>>>> Arun
>>>>>>>
>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>> acm@hortonworks.com
>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Folks,
>>>>>>>>
>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>> to
>>>>>>>> hadoop-2.6.
>>>>>>>>
>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>
>>>>>>>>        • HADOOP
>>>>>>>>                • Credential provider HADOOP-10607
>>>>>>>>        • HDFS
>>>>>>>>                • Heterogeneous storage (Phase 2) - Support APIs
>>> for
>>>>>>> using
>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>                • Memory as storage tier HDFS-5851
>>>>>>>>        • YARN
>>>>>>>>                • Dynamic Resource Configuration YARN-291
>>>>>>>>                • NodeManager Restart YARN-1336
>>>>>>>>                • ResourceManager HA Phase 2 YARN-556
>>>>>>>>                • Support for admin-specified labels in YARN
>>>> YARN-796
>>>>>>>>                • Support for automatic, shared cache for YARN
>>>>>>> application
>>>>>>>> artifacts YARN-1492
>>>>>>>>                • Support NodeGroup layer topology on YARN
>> YARN-18
>>>>>>>>                • Support for Docker containers in YARN
>> YARN-1964
>>>>>>>>                • YARN service registry YARN-913
>>>>>>>>
>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>> won't.
>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>> would
>>>>>> be
>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>> up a
>>>>>>> cadence.
>>>>>>>>
>>>>>>>> More importantly, as we discussed previously, we'd like
>>> hadoop-2.6
>>>> to
>>>>>> be
>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>> start
>>>> a
>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>> and
>>>>>> see
>>>>>>>> how they feel about this.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Arun
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> --
>>>>>>> Arun C. Murthy
>>>>>>> Hortonworks Inc.
>>>>>>> http://hortonworks.com/
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>> entity
>>>> to
>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>> of this message is not the intended recipient, you are hereby notified
>>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>> immediately
>>>>> and delete it from your system. Thank You.
>>
>>
>>
>> --
>> http://hortonworks.com/download/
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun Murthy <ac...@hortonworks.com>.

Sounds good. I'll branch this weekend and we can merge the jiras we
discussed in this thread as they they get wrapped next week.

Thanks everyone.

Arun


> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> We can branch off in a week or two so that work on branch-2 itself can go
> ahead with other features that can't fit in 2.6. Independent of that, we
> can then decide on the timeline of the release candidates once branch-2.6
> is close to being done w.r.t the planned features.
>
> Branching it off can let us focus on specific features that we want in for
> 2.6 and then eventually blockers for the release, nothing else. There is a
> trivial pain of committing to one more branch, but it's worth it in this
> case IMO.
>
> A lot of efforts are happening in parallel from the YARN side from where I
> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
> don't branch off and selectively try to get stuff in, it is likely to be in
> a perpetual delay.
>
> My 2 cents.
>
> +Vinod
>
> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
>> Given some of the features are in final stages of stabilization,
>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>> All the features in flux are important ones and worth delaying the release
>> by a week.
>>
>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>>
>>> Hey Nicholas,
>>>
>>> My concern about Archival Storage isn't related to the code quality or
>> the
>>> size of the feature. I think that you and Jing did good work. My concern
>> is
>>> that once we ship, we're locked into that set of archival storage APIs,
>> and
>>> these APIs are not yet finalized. Simply being able to turn off the
>> feature
>>> does not change the compatibility story.
>>>
>>> I'm willing to devote time to help review these JIRAs and kick the tires
>> on
>>> the APIs, but my point above was that I'm not sure it'd all be done by
>> the
>>> end of the week. Testing might also reveal additional changes that need
>> to
>>> be made, which also might not happen by end-of-week.
>>>
>>> I guess the question before us is if we're comfortable putting something
>> in
>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>> that as long as we're all aware that this might happen.
>>>
>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>> fine
>>> with it's inclusion, just a heads up that we might need some extra time
>> to
>>> finalize things before an RC can be cut.
>>>
>>> Thanks,
>>> Andrew
>>>
>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am worry about KMS and transparent encryption since there are quite
>>> many
>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>> impression
>>>> that the feature is not yet well tested.  Indeed, transparent
>> encryption
>>> is
>>>> a complicated feature which changes the core part of HDFS.  It is not
>>> easy
>>>> to get everything right.
>>>>
>>>>
>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>> block
>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>> storage,
>>>> the blocks will be stored using the appropriate storage types specified
>>> by
>>>> storage policies assigned to the files/directories.  Cluster admin
>> could
>>>> disable the feature by simply not configuring any storage type and not
>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>> is
>>>> in the final stages to be merged to branch-2.
>>>>
>>>> Regards,
>>>> Tsz-Wo
>>>>
>>>>
>>>>
>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>> suresh@hortonworks.com> wrote:
>>>>
>>>>
>>>>>
>>>>>
>>>>> I actually would like to see both archival storage and single replica
>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>> stages
>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>> on
>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>> these
>>>>> features are being in development for sometime.
>>>>>
>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>> andrew.wang@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Hey Arun,
>>>>>>
>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>> add/retarget
>>>>>> things accordingly?
>>>>>>
>>>>>> I think the KMS and transparent encryption are ready to go. We've
>> got
>>> a
>>>>>> very few further bug fixes pending, but that's it.
>>>>>>
>>>>>> Two HDFS things that I think probably won't make the end of the week
>>> are
>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>> (HDFS-6581),
>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>> to
>>>>>> trunk and I think needs a little more work before it goes into
>>> branch-2.
>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>> further
>>>> off
>>>>>> yet.
>>>>>>
>>>>>> Just my 2c as I did not work directly on these features. I just
>>>> generally
>>>>>> shy away from shipping bits quite this fresh.
>>>>>>
>>>>>> Thanks,
>>>>>> Andrew
>>>>>>
>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>> wrote:
>>>>>>
>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>> nicely.
>>>>>>>
>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>> to
>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> thanks,
>>>>>>> Arun
>>>>>>>
>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>> acm@hortonworks.com
>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Folks,
>>>>>>>>
>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>> to
>>>>>>>> hadoop-2.6.
>>>>>>>>
>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>
>>>>>>>>        • HADOOP
>>>>>>>>                • Credential provider HADOOP-10607
>>>>>>>>        • HDFS
>>>>>>>>                • Heterogeneous storage (Phase 2) - Support APIs
>>> for
>>>>>>> using
>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>                • Memory as storage tier HDFS-5851
>>>>>>>>        • YARN
>>>>>>>>                • Dynamic Resource Configuration YARN-291
>>>>>>>>                • NodeManager Restart YARN-1336
>>>>>>>>                • ResourceManager HA Phase 2 YARN-556
>>>>>>>>                • Support for admin-specified labels in YARN
>>>> YARN-796
>>>>>>>>                • Support for automatic, shared cache for YARN
>>>>>>> application
>>>>>>>> artifacts YARN-1492
>>>>>>>>                • Support NodeGroup layer topology on YARN
>> YARN-18
>>>>>>>>                • Support for Docker containers in YARN
>> YARN-1964
>>>>>>>>                • YARN service registry YARN-913
>>>>>>>>
>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>> won't.
>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>> would
>>>>>> be
>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>> up a
>>>>>>> cadence.
>>>>>>>>
>>>>>>>> More importantly, as we discussed previously, we'd like
>>> hadoop-2.6
>>>> to
>>>>>> be
>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>> start
>>>> a
>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>> and
>>>>>> see
>>>>>>>> how they feel about this.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Arun
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> --
>>>>>>> Arun C. Murthy
>>>>>>> Hortonworks Inc.
>>>>>>> http://hortonworks.com/
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>> entity
>>>> to
>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>> of this message is not the intended recipient, you are hereby notified
>>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>> immediately
>>>>> and delete it from your system. Thank You.
>>
>>
>>
>> --
>> http://hortonworks.com/download/
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun Murthy <ac...@hortonworks.com>.

Sounds good. I'll branch this weekend and we can merge the jiras we
discussed in this thread as they they get wrapped next week.

Thanks everyone.

Arun


> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> We can branch off in a week or two so that work on branch-2 itself can go
> ahead with other features that can't fit in 2.6. Independent of that, we
> can then decide on the timeline of the release candidates once branch-2.6
> is close to being done w.r.t the planned features.
>
> Branching it off can let us focus on specific features that we want in for
> 2.6 and then eventually blockers for the release, nothing else. There is a
> trivial pain of committing to one more branch, but it's worth it in this
> case IMO.
>
> A lot of efforts are happening in parallel from the YARN side from where I
> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
> don't branch off and selectively try to get stuff in, it is likely to be in
> a perpetual delay.
>
> My 2 cents.
>
> +Vinod
>
> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
>> Given some of the features are in final stages of stabilization,
>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>> All the features in flux are important ones and worth delaying the release
>> by a week.
>>
>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>>
>>> Hey Nicholas,
>>>
>>> My concern about Archival Storage isn't related to the code quality or
>> the
>>> size of the feature. I think that you and Jing did good work. My concern
>> is
>>> that once we ship, we're locked into that set of archival storage APIs,
>> and
>>> these APIs are not yet finalized. Simply being able to turn off the
>> feature
>>> does not change the compatibility story.
>>>
>>> I'm willing to devote time to help review these JIRAs and kick the tires
>> on
>>> the APIs, but my point above was that I'm not sure it'd all be done by
>> the
>>> end of the week. Testing might also reveal additional changes that need
>> to
>>> be made, which also might not happen by end-of-week.
>>>
>>> I guess the question before us is if we're comfortable putting something
>> in
>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>> that as long as we're all aware that this might happen.
>>>
>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>> fine
>>> with it's inclusion, just a heads up that we might need some extra time
>> to
>>> finalize things before an RC can be cut.
>>>
>>> Thanks,
>>> Andrew
>>>
>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am worry about KMS and transparent encryption since there are quite
>>> many
>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>> impression
>>>> that the feature is not yet well tested.  Indeed, transparent
>> encryption
>>> is
>>>> a complicated feature which changes the core part of HDFS.  It is not
>>> easy
>>>> to get everything right.
>>>>
>>>>
>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>> block
>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>> storage,
>>>> the blocks will be stored using the appropriate storage types specified
>>> by
>>>> storage policies assigned to the files/directories.  Cluster admin
>> could
>>>> disable the feature by simply not configuring any storage type and not
>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>> is
>>>> in the final stages to be merged to branch-2.
>>>>
>>>> Regards,
>>>> Tsz-Wo
>>>>
>>>>
>>>>
>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>> suresh@hortonworks.com> wrote:
>>>>
>>>>
>>>>>
>>>>>
>>>>> I actually would like to see both archival storage and single replica
>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>> stages
>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>> on
>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>> these
>>>>> features are being in development for sometime.
>>>>>
>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>> andrew.wang@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Hey Arun,
>>>>>>
>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>> add/retarget
>>>>>> things accordingly?
>>>>>>
>>>>>> I think the KMS and transparent encryption are ready to go. We've
>> got
>>> a
>>>>>> very few further bug fixes pending, but that's it.
>>>>>>
>>>>>> Two HDFS things that I think probably won't make the end of the week
>>> are
>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>> (HDFS-6581),
>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>> to
>>>>>> trunk and I think needs a little more work before it goes into
>>> branch-2.
>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>> further
>>>> off
>>>>>> yet.
>>>>>>
>>>>>> Just my 2c as I did not work directly on these features. I just
>>>> generally
>>>>>> shy away from shipping bits quite this fresh.
>>>>>>
>>>>>> Thanks,
>>>>>> Andrew
>>>>>>
>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>> wrote:
>>>>>>
>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>> nicely.
>>>>>>>
>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>> to
>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> thanks,
>>>>>>> Arun
>>>>>>>
>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>> acm@hortonworks.com
>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Folks,
>>>>>>>>
>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>> to
>>>>>>>> hadoop-2.6.
>>>>>>>>
>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>
>>>>>>>>        • HADOOP
>>>>>>>>                • Credential provider HADOOP-10607
>>>>>>>>        • HDFS
>>>>>>>>                • Heterogeneous storage (Phase 2) - Support APIs
>>> for
>>>>>>> using
>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>                • Memory as storage tier HDFS-5851
>>>>>>>>        • YARN
>>>>>>>>                • Dynamic Resource Configuration YARN-291
>>>>>>>>                • NodeManager Restart YARN-1336
>>>>>>>>                • ResourceManager HA Phase 2 YARN-556
>>>>>>>>                • Support for admin-specified labels in YARN
>>>> YARN-796
>>>>>>>>                • Support for automatic, shared cache for YARN
>>>>>>> application
>>>>>>>> artifacts YARN-1492
>>>>>>>>                • Support NodeGroup layer topology on YARN
>> YARN-18
>>>>>>>>                • Support for Docker containers in YARN
>> YARN-1964
>>>>>>>>                • YARN service registry YARN-913
>>>>>>>>
>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>> won't.
>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>> would
>>>>>> be
>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>> up a
>>>>>>> cadence.
>>>>>>>>
>>>>>>>> More importantly, as we discussed previously, we'd like
>>> hadoop-2.6
>>>> to
>>>>>> be
>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>> start
>>>> a
>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>> and
>>>>>> see
>>>>>>>> how they feel about this.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Arun
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> --
>>>>>>> Arun C. Murthy
>>>>>>> Hortonworks Inc.
>>>>>>> http://hortonworks.com/
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>> entity
>>>> to
>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>> of this message is not the intended recipient, you are hereby notified
>>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>> immediately
>>>>> and delete it from your system. Thank You.
>>
>>
>>
>> --
>> http://hortonworks.com/download/
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun Murthy <ac...@hortonworks.com>.

Sounds good. I'll branch this weekend and we can merge the jiras we
discussed in this thread as they they get wrapped next week.

Thanks everyone.

Arun


> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> We can branch off in a week or two so that work on branch-2 itself can go
> ahead with other features that can't fit in 2.6. Independent of that, we
> can then decide on the timeline of the release candidates once branch-2.6
> is close to being done w.r.t the planned features.
>
> Branching it off can let us focus on specific features that we want in for
> 2.6 and then eventually blockers for the release, nothing else. There is a
> trivial pain of committing to one more branch, but it's worth it in this
> case IMO.
>
> A lot of efforts are happening in parallel from the YARN side from where I
> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
> don't branch off and selectively try to get stuff in, it is likely to be in
> a perpetual delay.
>
> My 2 cents.
>
> +Vinod
>
> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
>> Given some of the features are in final stages of stabilization,
>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>> All the features in flux are important ones and worth delaying the release
>> by a week.
>>
>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>>
>>> Hey Nicholas,
>>>
>>> My concern about Archival Storage isn't related to the code quality or
>> the
>>> size of the feature. I think that you and Jing did good work. My concern
>> is
>>> that once we ship, we're locked into that set of archival storage APIs,
>> and
>>> these APIs are not yet finalized. Simply being able to turn off the
>> feature
>>> does not change the compatibility story.
>>>
>>> I'm willing to devote time to help review these JIRAs and kick the tires
>> on
>>> the APIs, but my point above was that I'm not sure it'd all be done by
>> the
>>> end of the week. Testing might also reveal additional changes that need
>> to
>>> be made, which also might not happen by end-of-week.
>>>
>>> I guess the question before us is if we're comfortable putting something
>> in
>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>> that as long as we're all aware that this might happen.
>>>
>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>> fine
>>> with it's inclusion, just a heads up that we might need some extra time
>> to
>>> finalize things before an RC can be cut.
>>>
>>> Thanks,
>>> Andrew
>>>
>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am worry about KMS and transparent encryption since there are quite
>>> many
>>>> bugs discovered after it got merged to branch-2.  It gives us an
>>> impression
>>>> that the feature is not yet well tested.  Indeed, transparent
>> encryption
>>> is
>>>> a complicated feature which changes the core part of HDFS.  It is not
>>> easy
>>>> to get everything right.
>>>>
>>>>
>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>> feature.  It introduces a new storage type ARCHIVE and the concept of
>>> block
>>>> storage policy to HDFS.  When a cluster is configured with ARCHIVE
>>> storage,
>>>> the blocks will be stored using the appropriate storage types specified
>>> by
>>>> storage policies assigned to the files/directories.  Cluster admin
>> could
>>>> disable the feature by simply not configuring any storage type and not
>>>> setting any storage policy as before.   As Suresh mentioned, HDFS-6584
>> is
>>>> in the final stages to be merged to branch-2.
>>>>
>>>> Regards,
>>>> Tsz-Wo
>>>>
>>>>
>>>>
>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>> suresh@hortonworks.com> wrote:
>>>>
>>>>
>>>>>
>>>>>
>>>>> I actually would like to see both archival storage and single replica
>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>> stages
>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>> on
>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>> these
>>>>> features are being in development for sometime.
>>>>>
>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>> andrew.wang@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Hey Arun,
>>>>>>
>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>> add/retarget
>>>>>> things accordingly?
>>>>>>
>>>>>> I think the KMS and transparent encryption are ready to go. We've
>> got
>>> a
>>>>>> very few further bug fixes pending, but that's it.
>>>>>>
>>>>>> Two HDFS things that I think probably won't make the end of the week
>>> are
>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>> (HDFS-6581),
>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>> to
>>>>>> trunk and I think needs a little more work before it goes into
>>> branch-2.
>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>> further
>>>> off
>>>>>> yet.
>>>>>>
>>>>>> Just my 2c as I did not work directly on these features. I just
>>>> generally
>>>>>> shy away from shipping bits quite this fresh.
>>>>>>
>>>>>> Thanks,
>>>>>> Andrew
>>>>>>
>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>> wrote:
>>>>>>
>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>> nicely.
>>>>>>>
>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>> to
>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> thanks,
>>>>>>> Arun
>>>>>>>
>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>> acm@hortonworks.com
>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Folks,
>>>>>>>>
>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>> to
>>>>>>>> hadoop-2.6.
>>>>>>>>
>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>
>>>>>>>>        • HADOOP
>>>>>>>>                • Credential provider HADOOP-10607
>>>>>>>>        • HDFS
>>>>>>>>                • Heterogeneous storage (Phase 2) - Support APIs
>>> for
>>>>>>> using
>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>                • Memory as storage tier HDFS-5851
>>>>>>>>        • YARN
>>>>>>>>                • Dynamic Resource Configuration YARN-291
>>>>>>>>                • NodeManager Restart YARN-1336
>>>>>>>>                • ResourceManager HA Phase 2 YARN-556
>>>>>>>>                • Support for admin-specified labels in YARN
>>>> YARN-796
>>>>>>>>                • Support for automatic, shared cache for YARN
>>>>>>> application
>>>>>>>> artifacts YARN-1492
>>>>>>>>                • Support NodeGroup layer topology on YARN
>> YARN-18
>>>>>>>>                • Support for Docker containers in YARN
>> YARN-1964
>>>>>>>>                • YARN service registry YARN-913
>>>>>>>>
>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>> won't.
>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>> would
>>>>>> be
>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>> up a
>>>>>>> cadence.
>>>>>>>>
>>>>>>>> More importantly, as we discussed previously, we'd like
>>> hadoop-2.6
>>>> to
>>>>>> be
>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>> start
>>>> a
>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>> and
>>>>>> see
>>>>>>>> how they feel about this.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Arun
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> --
>>>>>>> Arun C. Murthy
>>>>>>> Hortonworks Inc.
>>>>>>> http://hortonworks.com/
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>> entity
>>>> to
>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>> of this message is not the intended recipient, you are hereby notified
>>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>> immediately
>>>>> and delete it from your system. Thank You.
>>
>>
>>
>> --
>> http://hortonworks.com/download/
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.

We can branch off in a week or two so that work on branch-2 itself can go
ahead with other features that can't fit in 2.6. Independent of that, we
can then decide on the timeline of the release candidates once branch-2.6
is close to being done w.r.t the planned features.

Branching it off can let us focus on specific features that we want in for
2.6 and then eventually blockers for the release, nothing else. There is a
trivial pain of committing to one more branch, but it's worth it in this
case IMO.

A lot of efforts are happening in parallel from the YARN side from where I
see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
don't branch off and selectively try to get stuff in, it is likely to be in
a perpetual delay.

My 2 cents.

+Vinod

On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> Given some of the features are in final stages of stabilization,
> Arun, we should hold off creating 2.6 branch or building an RC by a week?
> All the features in flux are important ones and worth delaying the release
> by a week.
>
> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Nicholas,
> >
> > My concern about Archival Storage isn't related to the code quality or
> the
> > size of the feature. I think that you and Jing did good work. My concern
> is
> > that once we ship, we're locked into that set of archival storage APIs,
> and
> > these APIs are not yet finalized. Simply being able to turn off the
> feature
> > does not change the compatibility story.
> >
> > I'm willing to devote time to help review these JIRAs and kick the tires
> on
> > the APIs, but my point above was that I'm not sure it'd all be done by
> the
> > end of the week. Testing might also reveal additional changes that need
> to
> > be made, which also might not happen by end-of-week.
> >
> > I guess the question before us is if we're comfortable putting something
> in
> > branch-2.6 and then potentially adding API changes after. I'm okay with
> > that as long as we're all aware that this might happen.
> >
> > Arun, as RM is this cool with you? Again, I like this feature and I'm
> fine
> > with it's inclusion, just a heads up that we might need some extra time
> to
> > finalize things before an RC can be cut.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> > s29752-hadoopdev@yahoo.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > I am worry about KMS and transparent encryption since there are quite
> > many
> > > bugs discovered after it got merged to branch-2.  It gives us an
> > impression
> > > that the feature is not yet well tested.  Indeed, transparent
> encryption
> > is
> > > a complicated feature which changes the core part of HDFS.  It is not
> > easy
> > > to get everything right.
> > >
> > >
> > > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > > feature.  It introduces a new storage type ARCHIVE and the concept of
> > block
> > > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> > storage,
> > > the blocks will be stored using the appropriate storage types specified
> > by
> > > storage policies assigned to the files/directories.  Cluster admin
> could
> > > disable the feature by simply not configuring any storage type and not
> > > setting any storage policy as before.   As Suresh mentioned, HDFS-6584
> is
> > > in the final stages to be merged to branch-2.
> > >
> > > Regards,
> > > Tsz-Wo
> > >
> > >
> > >
> > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > > suresh@hortonworks.com> wrote:
> > >
> > >
> > > >
> > > >
> > > >I actually would like to see both archival storage and single replica
> > > >memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > >of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > >features are being in development for sometime.
> > > >
> > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > >wrote:
> > > >
> > > >> Hey Arun,
> > > >>
> > > >> Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > >> things accordingly?
> > > >>
> > > >> I think the KMS and transparent encryption are ready to go. We've
> got
> > a
> > > >> very few further bug fixes pending, but that's it.
> > > >>
> > > >> Two HDFS things that I think probably won't make the end of the week
> > are
> > > >> archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > >> which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > >> trunk and I think needs a little more work before it goes into
> > branch-2.
> > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > >> yet.
> > > >>
> > > >> Just my 2c as I did not work directly on these features. I just
> > > generally
> > > >> shy away from shipping bits quite this fresh.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >>
> > > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > >> >
> > > >> > I'll create branch-2.6 by end of the week and we can go from there
> > to
> > > >> > stabilize it - hopefully in the next few weeks.
> > > >> >
> > > >> > Thoughts?
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Folks,
> > > >> > >
> > > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > >> > > hadoop-2.6.
> > > >> > >
> > > >> > >  Currently, here is the Roadmap per the wiki:
> > > >> > >
> > > >> > >         • HADOOP
> > > >> > >                 • Credential provider HADOOP-10607
> > > >> > >         • HDFS
> > > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > >> > using
> > > >> > > storage tiers by the applications HDFS-5682
> > > >> > >                 • Memory as storage tier HDFS-5851
> > > >> > >         • YARN
> > > >> > >                 • Dynamic Resource Configuration YARN-291
> > > >> > >                 • NodeManager Restart YARN-1336
> > > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > > >> > >                 • Support for admin-specified labels in YARN
> > > YARN-796
> > > >> > >                 • Support for automatic, shared cache for YARN
> > > >> > application
> > > >> > > artifacts YARN-1492
> > > >> > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > >> > >                 • Support for Docker containers in YARN
> YARN-1964
> > > >> > >                 • YARN service registry YARN-913
> > > >> > >
> > > >> > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > >> be
> > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up a
> > > >> > cadence.
> > > >> > >
> > > >> > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > to
> > > >> be
> > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start
> > > a
> > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > >> see
> > > >> > > how they feel about this.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > >> to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > >--
> > > >http://hortonworks.com/download/
> > > >
> > > >--
> > > >CONFIDENTIALITY NOTICE
> > > >NOTICE: This message is intended for the use of the individual or
> entity
> > > to
> > > >which it is addressed and may contain information that is
> confidential,
> > > >privileged and exempt from disclosure under applicable law. If the
> > reader
> > > >of this message is not the intended recipient, you are hereby notified
> > > that
> > > >any printing, copying, dissemination, distribution, disclosure or
> > > >forwarding of this communication is strictly prohibited. If you have
> > > >received this communication in error, please contact the sender
> > > immediately
> > > >and delete it from your system. Thank You.
> > > >
> > > >
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Jitendra Pandey <ji...@hortonworks.com>.

I also believe its worth a week's wait to include HDFS-6584 and HDFS-6581
in 2.6.

On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> Given some of the features are in final stages of stabilization,
> Arun, we should hold off creating 2.6 branch or building an RC by a week?
> All the features in flux are important ones and worth delaying the release
> by a week.
>
> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Nicholas,
> >
> > My concern about Archival Storage isn't related to the code quality or
> the
> > size of the feature. I think that you and Jing did good work. My concern
> is
> > that once we ship, we're locked into that set of archival storage APIs,
> and
> > these APIs are not yet finalized. Simply being able to turn off the
> feature
> > does not change the compatibility story.
> >
> > I'm willing to devote time to help review these JIRAs and kick the tires
> on
> > the APIs, but my point above was that I'm not sure it'd all be done by
> the
> > end of the week. Testing might also reveal additional changes that need
> to
> > be made, which also might not happen by end-of-week.
> >
> > I guess the question before us is if we're comfortable putting something
> in
> > branch-2.6 and then potentially adding API changes after. I'm okay with
> > that as long as we're all aware that this might happen.
> >
> > Arun, as RM is this cool with you? Again, I like this feature and I'm
> fine
> > with it's inclusion, just a heads up that we might need some extra time
> to
> > finalize things before an RC can be cut.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> > s29752-hadoopdev@yahoo.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > I am worry about KMS and transparent encryption since there are quite
> > many
> > > bugs discovered after it got merged to branch-2.  It gives us an
> > impression
> > > that the feature is not yet well tested.  Indeed, transparent
> encryption
> > is
> > > a complicated feature which changes the core part of HDFS.  It is not
> > easy
> > > to get everything right.
> > >
> > >
> > > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > > feature.  It introduces a new storage type ARCHIVE and the concept of
> > block
> > > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> > storage,
> > > the blocks will be stored using the appropriate storage types specified
> > by
> > > storage policies assigned to the files/directories.  Cluster admin
> could
> > > disable the feature by simply not configuring any storage type and not
> > > setting any storage policy as before.   As Suresh mentioned, HDFS-6584
> is
> > > in the final stages to be merged to branch-2.
> > >
> > > Regards,
> > > Tsz-Wo
> > >
> > >
> > >
> > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > > suresh@hortonworks.com> wrote:
> > >
> > >
> > > >
> > > >
> > > >I actually would like to see both archival storage and single replica
> > > >memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > >of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > >features are being in development for sometime.
> > > >
> > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > >wrote:
> > > >
> > > >> Hey Arun,
> > > >>
> > > >> Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > >> things accordingly?
> > > >>
> > > >> I think the KMS and transparent encryption are ready to go. We've
> got
> > a
> > > >> very few further bug fixes pending, but that's it.
> > > >>
> > > >> Two HDFS things that I think probably won't make the end of the week
> > are
> > > >> archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > >> which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > >> trunk and I think needs a little more work before it goes into
> > branch-2.
> > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > >> yet.
> > > >>
> > > >> Just my 2c as I did not work directly on these features. I just
> > > generally
> > > >> shy away from shipping bits quite this fresh.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >>
> > > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > >> >
> > > >> > I'll create branch-2.6 by end of the week and we can go from there
> > to
> > > >> > stabilize it - hopefully in the next few weeks.
> > > >> >
> > > >> > Thoughts?
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Folks,
> > > >> > >
> > > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > >> > > hadoop-2.6.
> > > >> > >
> > > >> > >  Currently, here is the Roadmap per the wiki:
> > > >> > >
> > > >> > >         • HADOOP
> > > >> > >                 • Credential provider HADOOP-10607
> > > >> > >         • HDFS
> > > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > >> > using
> > > >> > > storage tiers by the applications HDFS-5682
> > > >> > >                 • Memory as storage tier HDFS-5851
> > > >> > >         • YARN
> > > >> > >                 • Dynamic Resource Configuration YARN-291
> > > >> > >                 • NodeManager Restart YARN-1336
> > > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > > >> > >                 • Support for admin-specified labels in YARN
> > > YARN-796
> > > >> > >                 • Support for automatic, shared cache for YARN
> > > >> > application
> > > >> > > artifacts YARN-1492
> > > >> > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > >> > >                 • Support for Docker containers in YARN
> YARN-1964
> > > >> > >                 • YARN service registry YARN-913
> > > >> > >
> > > >> > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > >> be
> > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up a
> > > >> > cadence.
> > > >> > >
> > > >> > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > to
> > > >> be
> > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start
> > > a
> > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > >> see
> > > >> > > how they feel about this.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > >> to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > >--
> > > >http://hortonworks.com/download/
> > > >
> > > >--
> > > >CONFIDENTIALITY NOTICE
> > > >NOTICE: This message is intended for the use of the individual or
> entity
> > > to
> > > >which it is addressed and may contain information that is
> confidential,
> > > >privileged and exempt from disclosure under applicable law. If the
> > reader
> > > >of this message is not the intended recipient, you are hereby notified
> > > that
> > > >any printing, copying, dissemination, distribution, disclosure or
> > > >forwarding of this communication is strictly prohibited. If you have
> > > >received this communication in error, please contact the sender
> > > immediately
> > > >and delete it from your system. Thank You.
> > > >
> > > >
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
<http://hortonworks.com/download/>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.

We can branch off in a week or two so that work on branch-2 itself can go
ahead with other features that can't fit in 2.6. Independent of that, we
can then decide on the timeline of the release candidates once branch-2.6
is close to being done w.r.t the planned features.

Branching it off can let us focus on specific features that we want in for
2.6 and then eventually blockers for the release, nothing else. There is a
trivial pain of committing to one more branch, but it's worth it in this
case IMO.

A lot of efforts are happening in parallel from the YARN side from where I
see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
don't branch off and selectively try to get stuff in, it is likely to be in
a perpetual delay.

My 2 cents.

+Vinod

On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> Given some of the features are in final stages of stabilization,
> Arun, we should hold off creating 2.6 branch or building an RC by a week?
> All the features in flux are important ones and worth delaying the release
> by a week.
>
> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Nicholas,
> >
> > My concern about Archival Storage isn't related to the code quality or
> the
> > size of the feature. I think that you and Jing did good work. My concern
> is
> > that once we ship, we're locked into that set of archival storage APIs,
> and
> > these APIs are not yet finalized. Simply being able to turn off the
> feature
> > does not change the compatibility story.
> >
> > I'm willing to devote time to help review these JIRAs and kick the tires
> on
> > the APIs, but my point above was that I'm not sure it'd all be done by
> the
> > end of the week. Testing might also reveal additional changes that need
> to
> > be made, which also might not happen by end-of-week.
> >
> > I guess the question before us is if we're comfortable putting something
> in
> > branch-2.6 and then potentially adding API changes after. I'm okay with
> > that as long as we're all aware that this might happen.
> >
> > Arun, as RM is this cool with you? Again, I like this feature and I'm
> fine
> > with it's inclusion, just a heads up that we might need some extra time
> to
> > finalize things before an RC can be cut.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> > s29752-hadoopdev@yahoo.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > I am worry about KMS and transparent encryption since there are quite
> > many
> > > bugs discovered after it got merged to branch-2.  It gives us an
> > impression
> > > that the feature is not yet well tested.  Indeed, transparent
> encryption
> > is
> > > a complicated feature which changes the core part of HDFS.  It is not
> > easy
> > > to get everything right.
> > >
> > >
> > > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > > feature.  It introduces a new storage type ARCHIVE and the concept of
> > block
> > > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> > storage,
> > > the blocks will be stored using the appropriate storage types specified
> > by
> > > storage policies assigned to the files/directories.  Cluster admin
> could
> > > disable the feature by simply not configuring any storage type and not
> > > setting any storage policy as before.   As Suresh mentioned, HDFS-6584
> is
> > > in the final stages to be merged to branch-2.
> > >
> > > Regards,
> > > Tsz-Wo
> > >
> > >
> > >
> > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > > suresh@hortonworks.com> wrote:
> > >
> > >
> > > >
> > > >
> > > >I actually would like to see both archival storage and single replica
> > > >memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > >of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > >features are being in development for sometime.
> > > >
> > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > >wrote:
> > > >
> > > >> Hey Arun,
> > > >>
> > > >> Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > >> things accordingly?
> > > >>
> > > >> I think the KMS and transparent encryption are ready to go. We've
> got
> > a
> > > >> very few further bug fixes pending, but that's it.
> > > >>
> > > >> Two HDFS things that I think probably won't make the end of the week
> > are
> > > >> archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > >> which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > >> trunk and I think needs a little more work before it goes into
> > branch-2.
> > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > >> yet.
> > > >>
> > > >> Just my 2c as I did not work directly on these features. I just
> > > generally
> > > >> shy away from shipping bits quite this fresh.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >>
> > > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > >> >
> > > >> > I'll create branch-2.6 by end of the week and we can go from there
> > to
> > > >> > stabilize it - hopefully in the next few weeks.
> > > >> >
> > > >> > Thoughts?
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Folks,
> > > >> > >
> > > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > >> > > hadoop-2.6.
> > > >> > >
> > > >> > >  Currently, here is the Roadmap per the wiki:
> > > >> > >
> > > >> > >         • HADOOP
> > > >> > >                 • Credential provider HADOOP-10607
> > > >> > >         • HDFS
> > > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > >> > using
> > > >> > > storage tiers by the applications HDFS-5682
> > > >> > >                 • Memory as storage tier HDFS-5851
> > > >> > >         • YARN
> > > >> > >                 • Dynamic Resource Configuration YARN-291
> > > >> > >                 • NodeManager Restart YARN-1336
> > > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > > >> > >                 • Support for admin-specified labels in YARN
> > > YARN-796
> > > >> > >                 • Support for automatic, shared cache for YARN
> > > >> > application
> > > >> > > artifacts YARN-1492
> > > >> > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > >> > >                 • Support for Docker containers in YARN
> YARN-1964
> > > >> > >                 • YARN service registry YARN-913
> > > >> > >
> > > >> > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > >> be
> > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up a
> > > >> > cadence.
> > > >> > >
> > > >> > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > to
> > > >> be
> > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start
> > > a
> > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > >> see
> > > >> > > how they feel about this.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > >> to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > >--
> > > >http://hortonworks.com/download/
> > > >
> > > >--
> > > >CONFIDENTIALITY NOTICE
> > > >NOTICE: This message is intended for the use of the individual or
> entity
> > > to
> > > >which it is addressed and may contain information that is
> confidential,
> > > >privileged and exempt from disclosure under applicable law. If the
> > reader
> > > >of this message is not the intended recipient, you are hereby notified
> > > that
> > > >any printing, copying, dissemination, distribution, disclosure or
> > > >forwarding of this communication is strictly prohibited. If you have
> > > >received this communication in error, please contact the sender
> > > immediately
> > > >and delete it from your system. Thank You.
> > > >
> > > >
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Jitendra Pandey <ji...@hortonworks.com>.

I also believe its worth a week's wait to include HDFS-6584 and HDFS-6581
in 2.6.

On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> Given some of the features are in final stages of stabilization,
> Arun, we should hold off creating 2.6 branch or building an RC by a week?
> All the features in flux are important ones and worth delaying the release
> by a week.
>
> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Nicholas,
> >
> > My concern about Archival Storage isn't related to the code quality or
> the
> > size of the feature. I think that you and Jing did good work. My concern
> is
> > that once we ship, we're locked into that set of archival storage APIs,
> and
> > these APIs are not yet finalized. Simply being able to turn off the
> feature
> > does not change the compatibility story.
> >
> > I'm willing to devote time to help review these JIRAs and kick the tires
> on
> > the APIs, but my point above was that I'm not sure it'd all be done by
> the
> > end of the week. Testing might also reveal additional changes that need
> to
> > be made, which also might not happen by end-of-week.
> >
> > I guess the question before us is if we're comfortable putting something
> in
> > branch-2.6 and then potentially adding API changes after. I'm okay with
> > that as long as we're all aware that this might happen.
> >
> > Arun, as RM is this cool with you? Again, I like this feature and I'm
> fine
> > with it's inclusion, just a heads up that we might need some extra time
> to
> > finalize things before an RC can be cut.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> > s29752-hadoopdev@yahoo.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > I am worry about KMS and transparent encryption since there are quite
> > many
> > > bugs discovered after it got merged to branch-2.  It gives us an
> > impression
> > > that the feature is not yet well tested.  Indeed, transparent
> encryption
> > is
> > > a complicated feature which changes the core part of HDFS.  It is not
> > easy
> > > to get everything right.
> > >
> > >
> > > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > > feature.  It introduces a new storage type ARCHIVE and the concept of
> > block
> > > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> > storage,
> > > the blocks will be stored using the appropriate storage types specified
> > by
> > > storage policies assigned to the files/directories.  Cluster admin
> could
> > > disable the feature by simply not configuring any storage type and not
> > > setting any storage policy as before.   As Suresh mentioned, HDFS-6584
> is
> > > in the final stages to be merged to branch-2.
> > >
> > > Regards,
> > > Tsz-Wo
> > >
> > >
> > >
> > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > > suresh@hortonworks.com> wrote:
> > >
> > >
> > > >
> > > >
> > > >I actually would like to see both archival storage and single replica
> > > >memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > >of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > >features are being in development for sometime.
> > > >
> > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > >wrote:
> > > >
> > > >> Hey Arun,
> > > >>
> > > >> Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > >> things accordingly?
> > > >>
> > > >> I think the KMS and transparent encryption are ready to go. We've
> got
> > a
> > > >> very few further bug fixes pending, but that's it.
> > > >>
> > > >> Two HDFS things that I think probably won't make the end of the week
> > are
> > > >> archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > >> which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > >> trunk and I think needs a little more work before it goes into
> > branch-2.
> > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > >> yet.
> > > >>
> > > >> Just my 2c as I did not work directly on these features. I just
> > > generally
> > > >> shy away from shipping bits quite this fresh.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >>
> > > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > >> >
> > > >> > I'll create branch-2.6 by end of the week and we can go from there
> > to
> > > >> > stabilize it - hopefully in the next few weeks.
> > > >> >
> > > >> > Thoughts?
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Folks,
> > > >> > >
> > > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > >> > > hadoop-2.6.
> > > >> > >
> > > >> > >  Currently, here is the Roadmap per the wiki:
> > > >> > >
> > > >> > >         • HADOOP
> > > >> > >                 • Credential provider HADOOP-10607
> > > >> > >         • HDFS
> > > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > >> > using
> > > >> > > storage tiers by the applications HDFS-5682
> > > >> > >                 • Memory as storage tier HDFS-5851
> > > >> > >         • YARN
> > > >> > >                 • Dynamic Resource Configuration YARN-291
> > > >> > >                 • NodeManager Restart YARN-1336
> > > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > > >> > >                 • Support for admin-specified labels in YARN
> > > YARN-796
> > > >> > >                 • Support for automatic, shared cache for YARN
> > > >> > application
> > > >> > > artifacts YARN-1492
> > > >> > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > >> > >                 • Support for Docker containers in YARN
> YARN-1964
> > > >> > >                 • YARN service registry YARN-913
> > > >> > >
> > > >> > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > >> be
> > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up a
> > > >> > cadence.
> > > >> > >
> > > >> > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > to
> > > >> be
> > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start
> > > a
> > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > >> see
> > > >> > > how they feel about this.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > >> to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > >--
> > > >http://hortonworks.com/download/
> > > >
> > > >--
> > > >CONFIDENTIALITY NOTICE
> > > >NOTICE: This message is intended for the use of the individual or
> entity
> > > to
> > > >which it is addressed and may contain information that is
> confidential,
> > > >privileged and exempt from disclosure under applicable law. If the
> > reader
> > > >of this message is not the intended recipient, you are hereby notified
> > > that
> > > >any printing, copying, dissemination, distribution, disclosure or
> > > >forwarding of this communication is strictly prohibited. If you have
> > > >received this communication in error, please contact the sender
> > > immediately
> > > >and delete it from your system. Thank You.
> > > >
> > > >
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
<http://hortonworks.com/download/>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.

We can branch off in a week or two so that work on branch-2 itself can go
ahead with other features that can't fit in 2.6. Independent of that, we
can then decide on the timeline of the release candidates once branch-2.6
is close to being done w.r.t the planned features.

Branching it off can let us focus on specific features that we want in for
2.6 and then eventually blockers for the release, nothing else. There is a
trivial pain of committing to one more branch, but it's worth it in this
case IMO.

A lot of efforts are happening in parallel from the YARN side from where I
see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
don't branch off and selectively try to get stuff in, it is likely to be in
a perpetual delay.

My 2 cents.

+Vinod

On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> Given some of the features are in final stages of stabilization,
> Arun, we should hold off creating 2.6 branch or building an RC by a week?
> All the features in flux are important ones and worth delaying the release
> by a week.
>
> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Nicholas,
> >
> > My concern about Archival Storage isn't related to the code quality or
> the
> > size of the feature. I think that you and Jing did good work. My concern
> is
> > that once we ship, we're locked into that set of archival storage APIs,
> and
> > these APIs are not yet finalized. Simply being able to turn off the
> feature
> > does not change the compatibility story.
> >
> > I'm willing to devote time to help review these JIRAs and kick the tires
> on
> > the APIs, but my point above was that I'm not sure it'd all be done by
> the
> > end of the week. Testing might also reveal additional changes that need
> to
> > be made, which also might not happen by end-of-week.
> >
> > I guess the question before us is if we're comfortable putting something
> in
> > branch-2.6 and then potentially adding API changes after. I'm okay with
> > that as long as we're all aware that this might happen.
> >
> > Arun, as RM is this cool with you? Again, I like this feature and I'm
> fine
> > with it's inclusion, just a heads up that we might need some extra time
> to
> > finalize things before an RC can be cut.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> > s29752-hadoopdev@yahoo.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > I am worry about KMS and transparent encryption since there are quite
> > many
> > > bugs discovered after it got merged to branch-2.  It gives us an
> > impression
> > > that the feature is not yet well tested.  Indeed, transparent
> encryption
> > is
> > > a complicated feature which changes the core part of HDFS.  It is not
> > easy
> > > to get everything right.
> > >
> > >
> > > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > > feature.  It introduces a new storage type ARCHIVE and the concept of
> > block
> > > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> > storage,
> > > the blocks will be stored using the appropriate storage types specified
> > by
> > > storage policies assigned to the files/directories.  Cluster admin
> could
> > > disable the feature by simply not configuring any storage type and not
> > > setting any storage policy as before.   As Suresh mentioned, HDFS-6584
> is
> > > in the final stages to be merged to branch-2.
> > >
> > > Regards,
> > > Tsz-Wo
> > >
> > >
> > >
> > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > > suresh@hortonworks.com> wrote:
> > >
> > >
> > > >
> > > >
> > > >I actually would like to see both archival storage and single replica
> > > >memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > >of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > >features are being in development for sometime.
> > > >
> > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > >wrote:
> > > >
> > > >> Hey Arun,
> > > >>
> > > >> Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > >> things accordingly?
> > > >>
> > > >> I think the KMS and transparent encryption are ready to go. We've
> got
> > a
> > > >> very few further bug fixes pending, but that's it.
> > > >>
> > > >> Two HDFS things that I think probably won't make the end of the week
> > are
> > > >> archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > >> which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > >> trunk and I think needs a little more work before it goes into
> > branch-2.
> > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > >> yet.
> > > >>
> > > >> Just my 2c as I did not work directly on these features. I just
> > > generally
> > > >> shy away from shipping bits quite this fresh.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >>
> > > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > >> >
> > > >> > I'll create branch-2.6 by end of the week and we can go from there
> > to
> > > >> > stabilize it - hopefully in the next few weeks.
> > > >> >
> > > >> > Thoughts?
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Folks,
> > > >> > >
> > > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > >> > > hadoop-2.6.
> > > >> > >
> > > >> > >  Currently, here is the Roadmap per the wiki:
> > > >> > >
> > > >> > >         • HADOOP
> > > >> > >                 • Credential provider HADOOP-10607
> > > >> > >         • HDFS
> > > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > >> > using
> > > >> > > storage tiers by the applications HDFS-5682
> > > >> > >                 • Memory as storage tier HDFS-5851
> > > >> > >         • YARN
> > > >> > >                 • Dynamic Resource Configuration YARN-291
> > > >> > >                 • NodeManager Restart YARN-1336
> > > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > > >> > >                 • Support for admin-specified labels in YARN
> > > YARN-796
> > > >> > >                 • Support for automatic, shared cache for YARN
> > > >> > application
> > > >> > > artifacts YARN-1492
> > > >> > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > >> > >                 • Support for Docker containers in YARN
> YARN-1964
> > > >> > >                 • YARN service registry YARN-913
> > > >> > >
> > > >> > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > >> be
> > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up a
> > > >> > cadence.
> > > >> > >
> > > >> > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > to
> > > >> be
> > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start
> > > a
> > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > >> see
> > > >> > > how they feel about this.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > >> to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > >--
> > > >http://hortonworks.com/download/
> > > >
> > > >--
> > > >CONFIDENTIALITY NOTICE
> > > >NOTICE: This message is intended for the use of the individual or
> entity
> > > to
> > > >which it is addressed and may contain information that is
> confidential,
> > > >privileged and exempt from disclosure under applicable law. If the
> > reader
> > > >of this message is not the intended recipient, you are hereby notified
> > > that
> > > >any printing, copying, dissemination, distribution, disclosure or
> > > >forwarding of this communication is strictly prohibited. If you have
> > > >received this communication in error, please contact the sender
> > > immediately
> > > >and delete it from your system. Thank You.
> > > >
> > > >
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Jitendra Pandey <ji...@hortonworks.com>.

I also believe its worth a week's wait to include HDFS-6584 and HDFS-6581
in 2.6.

On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> Given some of the features are in final stages of stabilization,
> Arun, we should hold off creating 2.6 branch or building an RC by a week?
> All the features in flux are important ones and worth delaying the release
> by a week.
>
> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Nicholas,
> >
> > My concern about Archival Storage isn't related to the code quality or
> the
> > size of the feature. I think that you and Jing did good work. My concern
> is
> > that once we ship, we're locked into that set of archival storage APIs,
> and
> > these APIs are not yet finalized. Simply being able to turn off the
> feature
> > does not change the compatibility story.
> >
> > I'm willing to devote time to help review these JIRAs and kick the tires
> on
> > the APIs, but my point above was that I'm not sure it'd all be done by
> the
> > end of the week. Testing might also reveal additional changes that need
> to
> > be made, which also might not happen by end-of-week.
> >
> > I guess the question before us is if we're comfortable putting something
> in
> > branch-2.6 and then potentially adding API changes after. I'm okay with
> > that as long as we're all aware that this might happen.
> >
> > Arun, as RM is this cool with you? Again, I like this feature and I'm
> fine
> > with it's inclusion, just a heads up that we might need some extra time
> to
> > finalize things before an RC can be cut.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> > s29752-hadoopdev@yahoo.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > I am worry about KMS and transparent encryption since there are quite
> > many
> > > bugs discovered after it got merged to branch-2.  It gives us an
> > impression
> > > that the feature is not yet well tested.  Indeed, transparent
> encryption
> > is
> > > a complicated feature which changes the core part of HDFS.  It is not
> > easy
> > > to get everything right.
> > >
> > >
> > > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > > feature.  It introduces a new storage type ARCHIVE and the concept of
> > block
> > > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> > storage,
> > > the blocks will be stored using the appropriate storage types specified
> > by
> > > storage policies assigned to the files/directories.  Cluster admin
> could
> > > disable the feature by simply not configuring any storage type and not
> > > setting any storage policy as before.   As Suresh mentioned, HDFS-6584
> is
> > > in the final stages to be merged to branch-2.
> > >
> > > Regards,
> > > Tsz-Wo
> > >
> > >
> > >
> > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > > suresh@hortonworks.com> wrote:
> > >
> > >
> > > >
> > > >
> > > >I actually would like to see both archival storage and single replica
> > > >memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > >of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > >features are being in development for sometime.
> > > >
> > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > >wrote:
> > > >
> > > >> Hey Arun,
> > > >>
> > > >> Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > >> things accordingly?
> > > >>
> > > >> I think the KMS and transparent encryption are ready to go. We've
> got
> > a
> > > >> very few further bug fixes pending, but that's it.
> > > >>
> > > >> Two HDFS things that I think probably won't make the end of the week
> > are
> > > >> archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > >> which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > >> trunk and I think needs a little more work before it goes into
> > branch-2.
> > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > >> yet.
> > > >>
> > > >> Just my 2c as I did not work directly on these features. I just
> > > generally
> > > >> shy away from shipping bits quite this fresh.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >>
> > > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > >> >
> > > >> > I'll create branch-2.6 by end of the week and we can go from there
> > to
> > > >> > stabilize it - hopefully in the next few weeks.
> > > >> >
> > > >> > Thoughts?
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Folks,
> > > >> > >
> > > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > >> > > hadoop-2.6.
> > > >> > >
> > > >> > >  Currently, here is the Roadmap per the wiki:
> > > >> > >
> > > >> > >         • HADOOP
> > > >> > >                 • Credential provider HADOOP-10607
> > > >> > >         • HDFS
> > > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > >> > using
> > > >> > > storage tiers by the applications HDFS-5682
> > > >> > >                 • Memory as storage tier HDFS-5851
> > > >> > >         • YARN
> > > >> > >                 • Dynamic Resource Configuration YARN-291
> > > >> > >                 • NodeManager Restart YARN-1336
> > > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > > >> > >                 • Support for admin-specified labels in YARN
> > > YARN-796
> > > >> > >                 • Support for automatic, shared cache for YARN
> > > >> > application
> > > >> > > artifacts YARN-1492
> > > >> > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > >> > >                 • Support for Docker containers in YARN
> YARN-1964
> > > >> > >                 • YARN service registry YARN-913
> > > >> > >
> > > >> > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > >> be
> > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up a
> > > >> > cadence.
> > > >> > >
> > > >> > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > to
> > > >> be
> > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start
> > > a
> > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > >> see
> > > >> > > how they feel about this.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > >> to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > >--
> > > >http://hortonworks.com/download/
> > > >
> > > >--
> > > >CONFIDENTIALITY NOTICE
> > > >NOTICE: This message is intended for the use of the individual or
> entity
> > > to
> > > >which it is addressed and may contain information that is
> confidential,
> > > >privileged and exempt from disclosure under applicable law. If the
> > reader
> > > >of this message is not the intended recipient, you are hereby notified
> > > that
> > > >any printing, copying, dissemination, distribution, disclosure or
> > > >forwarding of this communication is strictly prohibited. If you have
> > > >received this communication in error, please contact the sender
> > > immediately
> > > >and delete it from your system. Thank You.
> > > >
> > > >
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
<http://hortonworks.com/download/>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.

We can branch off in a week or two so that work on branch-2 itself can go
ahead with other features that can't fit in 2.6. Independent of that, we
can then decide on the timeline of the release candidates once branch-2.6
is close to being done w.r.t the planned features.

Branching it off can let us focus on specific features that we want in for
2.6 and then eventually blockers for the release, nothing else. There is a
trivial pain of committing to one more branch, but it's worth it in this
case IMO.

A lot of efforts are happening in parallel from the YARN side from where I
see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
don't branch off and selectively try to get stuff in, it is likely to be in
a perpetual delay.

My 2 cents.

+Vinod

On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> Given some of the features are in final stages of stabilization,
> Arun, we should hold off creating 2.6 branch or building an RC by a week?
> All the features in flux are important ones and worth delaying the release
> by a week.
>
> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Nicholas,
> >
> > My concern about Archival Storage isn't related to the code quality or
> the
> > size of the feature. I think that you and Jing did good work. My concern
> is
> > that once we ship, we're locked into that set of archival storage APIs,
> and
> > these APIs are not yet finalized. Simply being able to turn off the
> feature
> > does not change the compatibility story.
> >
> > I'm willing to devote time to help review these JIRAs and kick the tires
> on
> > the APIs, but my point above was that I'm not sure it'd all be done by
> the
> > end of the week. Testing might also reveal additional changes that need
> to
> > be made, which also might not happen by end-of-week.
> >
> > I guess the question before us is if we're comfortable putting something
> in
> > branch-2.6 and then potentially adding API changes after. I'm okay with
> > that as long as we're all aware that this might happen.
> >
> > Arun, as RM is this cool with you? Again, I like this feature and I'm
> fine
> > with it's inclusion, just a heads up that we might need some extra time
> to
> > finalize things before an RC can be cut.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> > s29752-hadoopdev@yahoo.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > I am worry about KMS and transparent encryption since there are quite
> > many
> > > bugs discovered after it got merged to branch-2.  It gives us an
> > impression
> > > that the feature is not yet well tested.  Indeed, transparent
> encryption
> > is
> > > a complicated feature which changes the core part of HDFS.  It is not
> > easy
> > > to get everything right.
> > >
> > >
> > > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > > feature.  It introduces a new storage type ARCHIVE and the concept of
> > block
> > > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> > storage,
> > > the blocks will be stored using the appropriate storage types specified
> > by
> > > storage policies assigned to the files/directories.  Cluster admin
> could
> > > disable the feature by simply not configuring any storage type and not
> > > setting any storage policy as before.   As Suresh mentioned, HDFS-6584
> is
> > > in the final stages to be merged to branch-2.
> > >
> > > Regards,
> > > Tsz-Wo
> > >
> > >
> > >
> > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > > suresh@hortonworks.com> wrote:
> > >
> > >
> > > >
> > > >
> > > >I actually would like to see both archival storage and single replica
> > > >memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > >of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > >features are being in development for sometime.
> > > >
> > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > >wrote:
> > > >
> > > >> Hey Arun,
> > > >>
> > > >> Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > >> things accordingly?
> > > >>
> > > >> I think the KMS and transparent encryption are ready to go. We've
> got
> > a
> > > >> very few further bug fixes pending, but that's it.
> > > >>
> > > >> Two HDFS things that I think probably won't make the end of the week
> > are
> > > >> archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > >> which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > >> trunk and I think needs a little more work before it goes into
> > branch-2.
> > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > >> yet.
> > > >>
> > > >> Just my 2c as I did not work directly on these features. I just
> > > generally
> > > >> shy away from shipping bits quite this fresh.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >>
> > > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > >> >
> > > >> > I'll create branch-2.6 by end of the week and we can go from there
> > to
> > > >> > stabilize it - hopefully in the next few weeks.
> > > >> >
> > > >> > Thoughts?
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Folks,
> > > >> > >
> > > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > >> > > hadoop-2.6.
> > > >> > >
> > > >> > >  Currently, here is the Roadmap per the wiki:
> > > >> > >
> > > >> > >         • HADOOP
> > > >> > >                 • Credential provider HADOOP-10607
> > > >> > >         • HDFS
> > > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > >> > using
> > > >> > > storage tiers by the applications HDFS-5682
> > > >> > >                 • Memory as storage tier HDFS-5851
> > > >> > >         • YARN
> > > >> > >                 • Dynamic Resource Configuration YARN-291
> > > >> > >                 • NodeManager Restart YARN-1336
> > > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > > >> > >                 • Support for admin-specified labels in YARN
> > > YARN-796
> > > >> > >                 • Support for automatic, shared cache for YARN
> > > >> > application
> > > >> > > artifacts YARN-1492
> > > >> > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > >> > >                 • Support for Docker containers in YARN
> YARN-1964
> > > >> > >                 • YARN service registry YARN-913
> > > >> > >
> > > >> > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > >> be
> > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up a
> > > >> > cadence.
> > > >> > >
> > > >> > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > to
> > > >> be
> > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start
> > > a
> > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > >> see
> > > >> > > how they feel about this.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > >> to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > >--
> > > >http://hortonworks.com/download/
> > > >
> > > >--
> > > >CONFIDENTIALITY NOTICE
> > > >NOTICE: This message is intended for the use of the individual or
> entity
> > > to
> > > >which it is addressed and may contain information that is
> confidential,
> > > >privileged and exempt from disclosure under applicable law. If the
> > reader
> > > >of this message is not the intended recipient, you are hereby notified
> > > that
> > > >any printing, copying, dissemination, distribution, disclosure or
> > > >forwarding of this communication is strictly prohibited. If you have
> > > >received this communication in error, please contact the sender
> > > immediately
> > > >and delete it from your system. Thank You.
> > > >
> > > >
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Suresh Srinivas <su...@hortonworks.com>.

Given some of the features are in final stages of stabilization,
Arun, we should hold off creating 2.6 branch or building an RC by a week?
All the features in flux are important ones and worth delaying the release
by a week.

On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
wrote:

> Hey Nicholas,
>
> My concern about Archival Storage isn't related to the code quality or the
> size of the feature. I think that you and Jing did good work. My concern is
> that once we ship, we're locked into that set of archival storage APIs, and
> these APIs are not yet finalized. Simply being able to turn off the feature
> does not change the compatibility story.
>
> I'm willing to devote time to help review these JIRAs and kick the tires on
> the APIs, but my point above was that I'm not sure it'd all be done by the
> end of the week. Testing might also reveal additional changes that need to
> be made, which also might not happen by end-of-week.
>
> I guess the question before us is if we're comfortable putting something in
> branch-2.6 and then potentially adding API changes after. I'm okay with
> that as long as we're all aware that this might happen.
>
> Arun, as RM is this cool with you? Again, I like this feature and I'm fine
> with it's inclusion, just a heads up that we might need some extra time to
> finalize things before an RC can be cut.
>
> Thanks,
> Andrew
>
> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> s29752-hadoopdev@yahoo.com.invalid> wrote:
>
> > Hi,
> >
> > I am worry about KMS and transparent encryption since there are quite
> many
> > bugs discovered after it got merged to branch-2.  It gives us an
> impression
> > that the feature is not yet well tested.  Indeed, transparent encryption
> is
> > a complicated feature which changes the core part of HDFS.  It is not
> easy
> > to get everything right.
> >
> >
> > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > feature.  It introduces a new storage type ARCHIVE and the concept of
> block
> > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> storage,
> > the blocks will be stored using the appropriate storage types specified
> by
> > storage policies assigned to the files/directories.  Cluster admin could
> > disable the feature by simply not configuring any storage type and not
> > setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is
> > in the final stages to be merged to branch-2.
> >
> > Regards,
> > Tsz-Wo
> >
> >
> >
> > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > suresh@hortonworks.com> wrote:
> >
> >
> > >
> > >
> > >I actually would like to see both archival storage and single replica
> > >memory writes to be in 2.6 release. Archival storage is in the final
> > stages
> > >of getting ready for branch-2 merge as Nicholas has already indicated on
> > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> these
> > >features are being in development for sometime.
> > >
> > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> > >wrote:
> > >
> > >> Hey Arun,
> > >>
> > >> Maybe we could do a quick run through of the Roadmap wiki and
> > add/retarget
> > >> things accordingly?
> > >>
> > >> I think the KMS and transparent encryption are ready to go. We've got
> a
> > >> very few further bug fixes pending, but that's it.
> > >>
> > >> Two HDFS things that I think probably won't make the end of the week
> are
> > >> archival storage (HDFS-6584) and single replica memory writes
> > (HDFS-6581),
> > >> which I believe are under the HSM banner. HDFS-6484 was just merged to
> > >> trunk and I think needs a little more work before it goes into
> branch-2.
> > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> > off
> > >> yet.
> > >>
> > >> Just my 2c as I did not work directly on these features. I just
> > generally
> > >> shy away from shipping bits quite this fresh.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > wrote:
> > >>
> > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > nicely.
> > >> >
> > >> > I'll create branch-2.6 by end of the week and we can go from there
> to
> > >> > stabilize it - hopefully in the next few weeks.
> > >> >
> > >> > Thoughts?
> > >> >
> > >> > thanks,
> > >> > Arun
> > >> >
> > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <acm@hortonworks.com
> >
> > >> > wrote:
> > >> >
> > >> > > Folks,
> > >> > >
> > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > >> > > hadoop-2.6.
> > >> > >
> > >> > >  Currently, here is the Roadmap per the wiki:
> > >> > >
> > >> > >         • HADOOP
> > >> > >                 • Credential provider HADOOP-10607
> > >> > >         • HDFS
> > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> for
> > >> > using
> > >> > > storage tiers by the applications HDFS-5682
> > >> > >                 • Memory as storage tier HDFS-5851
> > >> > >         • YARN
> > >> > >                 • Dynamic Resource Configuration YARN-291
> > >> > >                 • NodeManager Restart YARN-1336
> > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > >> > >                 • Support for admin-specified labels in YARN
> > YARN-796
> > >> > >                 • Support for automatic, shared cache for YARN
> > >> > application
> > >> > > artifacts YARN-1492
> > >> > >                 • Support NodeGroup layer topology on YARN YARN-18
> > >> > >                 • Support for Docker containers in YARN YARN-1964
> > >> > >                 • YARN service registry YARN-913
> > >> > >
> > >> > >  My suspicion is, as is normal, some will make the cut and some
> > won't.
> > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > would
> > >> be
> > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > >> > cadence.
> > >> > >
> > >> > >  More importantly, as we discussed previously, we'd like
> hadoop-2.6
> > to
> > >> be
> > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> start
> > a
> > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> and
> > >> see
> > >> > > how they feel about this.
> > >> > >
> > >> > > thanks,
> > >> > > Arun
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > --
> > >> > Arun C. Murthy
> > >> > Hortonworks Inc.
> > >> > http://hortonworks.com/
> > >> >
> > >> > --
> > >> > CONFIDENTIALITY NOTICE
> > >> > NOTICE: This message is intended for the use of the individual or
> > entity
> > >> to
> > >> > which it is addressed and may contain information that is
> > confidential,
> > >> > privileged and exempt from disclosure under applicable law. If the
> > reader
> > >> > of this message is not the intended recipient, you are hereby
> notified
> > >> that
> > >> > any printing, copying, dissemination, distribution, disclosure or
> > >> > forwarding of this communication is strictly prohibited. If you have
> > >> > received this communication in error, please contact the sender
> > >> immediately
> > >> > and delete it from your system. Thank You.
> > >> >
> > >>
> > >
> > >
> > >
> > >--
> > >http://hortonworks.com/download/
> > >
> > >--
> > >CONFIDENTIALITY NOTICE
> > >NOTICE: This message is intended for the use of the individual or entity
> > to
> > >which it is addressed and may contain information that is confidential,
> > >privileged and exempt from disclosure under applicable law. If the
> reader
> > >of this message is not the intended recipient, you are hereby notified
> > that
> > >any printing, copying, dissemination, distribution, disclosure or
> > >forwarding of this communication is strictly prohibited. If you have
> > >received this communication in error, please contact the sender
> > immediately
> > >and delete it from your system. Thank You.
> > >
> > >
> >
>



-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Suresh Srinivas <su...@hortonworks.com>.

Given some of the features are in final stages of stabilization,
Arun, we should hold off creating 2.6 branch or building an RC by a week?
All the features in flux are important ones and worth delaying the release
by a week.

On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
wrote:

> Hey Nicholas,
>
> My concern about Archival Storage isn't related to the code quality or the
> size of the feature. I think that you and Jing did good work. My concern is
> that once we ship, we're locked into that set of archival storage APIs, and
> these APIs are not yet finalized. Simply being able to turn off the feature
> does not change the compatibility story.
>
> I'm willing to devote time to help review these JIRAs and kick the tires on
> the APIs, but my point above was that I'm not sure it'd all be done by the
> end of the week. Testing might also reveal additional changes that need to
> be made, which also might not happen by end-of-week.
>
> I guess the question before us is if we're comfortable putting something in
> branch-2.6 and then potentially adding API changes after. I'm okay with
> that as long as we're all aware that this might happen.
>
> Arun, as RM is this cool with you? Again, I like this feature and I'm fine
> with it's inclusion, just a heads up that we might need some extra time to
> finalize things before an RC can be cut.
>
> Thanks,
> Andrew
>
> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> s29752-hadoopdev@yahoo.com.invalid> wrote:
>
> > Hi,
> >
> > I am worry about KMS and transparent encryption since there are quite
> many
> > bugs discovered after it got merged to branch-2.  It gives us an
> impression
> > that the feature is not yet well tested.  Indeed, transparent encryption
> is
> > a complicated feature which changes the core part of HDFS.  It is not
> easy
> > to get everything right.
> >
> >
> > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > feature.  It introduces a new storage type ARCHIVE and the concept of
> block
> > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> storage,
> > the blocks will be stored using the appropriate storage types specified
> by
> > storage policies assigned to the files/directories.  Cluster admin could
> > disable the feature by simply not configuring any storage type and not
> > setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is
> > in the final stages to be merged to branch-2.
> >
> > Regards,
> > Tsz-Wo
> >
> >
> >
> > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > suresh@hortonworks.com> wrote:
> >
> >
> > >
> > >
> > >I actually would like to see both archival storage and single replica
> > >memory writes to be in 2.6 release. Archival storage is in the final
> > stages
> > >of getting ready for branch-2 merge as Nicholas has already indicated on
> > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> these
> > >features are being in development for sometime.
> > >
> > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> > >wrote:
> > >
> > >> Hey Arun,
> > >>
> > >> Maybe we could do a quick run through of the Roadmap wiki and
> > add/retarget
> > >> things accordingly?
> > >>
> > >> I think the KMS and transparent encryption are ready to go. We've got
> a
> > >> very few further bug fixes pending, but that's it.
> > >>
> > >> Two HDFS things that I think probably won't make the end of the week
> are
> > >> archival storage (HDFS-6584) and single replica memory writes
> > (HDFS-6581),
> > >> which I believe are under the HSM banner. HDFS-6484 was just merged to
> > >> trunk and I think needs a little more work before it goes into
> branch-2.
> > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> > off
> > >> yet.
> > >>
> > >> Just my 2c as I did not work directly on these features. I just
> > generally
> > >> shy away from shipping bits quite this fresh.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > wrote:
> > >>
> > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > nicely.
> > >> >
> > >> > I'll create branch-2.6 by end of the week and we can go from there
> to
> > >> > stabilize it - hopefully in the next few weeks.
> > >> >
> > >> > Thoughts?
> > >> >
> > >> > thanks,
> > >> > Arun
> > >> >
> > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <acm@hortonworks.com
> >
> > >> > wrote:
> > >> >
> > >> > > Folks,
> > >> > >
> > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > >> > > hadoop-2.6.
> > >> > >
> > >> > >  Currently, here is the Roadmap per the wiki:
> > >> > >
> > >> > >         • HADOOP
> > >> > >                 • Credential provider HADOOP-10607
> > >> > >         • HDFS
> > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> for
> > >> > using
> > >> > > storage tiers by the applications HDFS-5682
> > >> > >                 • Memory as storage tier HDFS-5851
> > >> > >         • YARN
> > >> > >                 • Dynamic Resource Configuration YARN-291
> > >> > >                 • NodeManager Restart YARN-1336
> > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > >> > >                 • Support for admin-specified labels in YARN
> > YARN-796
> > >> > >                 • Support for automatic, shared cache for YARN
> > >> > application
> > >> > > artifacts YARN-1492
> > >> > >                 • Support NodeGroup layer topology on YARN YARN-18
> > >> > >                 • Support for Docker containers in YARN YARN-1964
> > >> > >                 • YARN service registry YARN-913
> > >> > >
> > >> > >  My suspicion is, as is normal, some will make the cut and some
> > won't.
> > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > would
> > >> be
> > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > >> > cadence.
> > >> > >
> > >> > >  More importantly, as we discussed previously, we'd like
> hadoop-2.6
> > to
> > >> be
> > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> start
> > a
> > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> and
> > >> see
> > >> > > how they feel about this.
> > >> > >
> > >> > > thanks,
> > >> > > Arun
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > --
> > >> > Arun C. Murthy
> > >> > Hortonworks Inc.
> > >> > http://hortonworks.com/
> > >> >
> > >> > --
> > >> > CONFIDENTIALITY NOTICE
> > >> > NOTICE: This message is intended for the use of the individual or
> > entity
> > >> to
> > >> > which it is addressed and may contain information that is
> > confidential,
> > >> > privileged and exempt from disclosure under applicable law. If the
> > reader
> > >> > of this message is not the intended recipient, you are hereby
> notified
> > >> that
> > >> > any printing, copying, dissemination, distribution, disclosure or
> > >> > forwarding of this communication is strictly prohibited. If you have
> > >> > received this communication in error, please contact the sender
> > >> immediately
> > >> > and delete it from your system. Thank You.
> > >> >
> > >>
> > >
> > >
> > >
> > >--
> > >http://hortonworks.com/download/
> > >
> > >--
> > >CONFIDENTIALITY NOTICE
> > >NOTICE: This message is intended for the use of the individual or entity
> > to
> > >which it is addressed and may contain information that is confidential,
> > >privileged and exempt from disclosure under applicable law. If the
> reader
> > >of this message is not the intended recipient, you are hereby notified
> > that
> > >any printing, copying, dissemination, distribution, disclosure or
> > >forwarding of this communication is strictly prohibited. If you have
> > >received this communication in error, please contact the sender
> > immediately
> > >and delete it from your system. Thank You.
> > >
> > >
> >
>



-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Suresh Srinivas <su...@hortonworks.com>.

Given some of the features are in final stages of stabilization,
Arun, we should hold off creating 2.6 branch or building an RC by a week?
All the features in flux are important ones and worth delaying the release
by a week.

On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
wrote:

> Hey Nicholas,
>
> My concern about Archival Storage isn't related to the code quality or the
> size of the feature. I think that you and Jing did good work. My concern is
> that once we ship, we're locked into that set of archival storage APIs, and
> these APIs are not yet finalized. Simply being able to turn off the feature
> does not change the compatibility story.
>
> I'm willing to devote time to help review these JIRAs and kick the tires on
> the APIs, but my point above was that I'm not sure it'd all be done by the
> end of the week. Testing might also reveal additional changes that need to
> be made, which also might not happen by end-of-week.
>
> I guess the question before us is if we're comfortable putting something in
> branch-2.6 and then potentially adding API changes after. I'm okay with
> that as long as we're all aware that this might happen.
>
> Arun, as RM is this cool with you? Again, I like this feature and I'm fine
> with it's inclusion, just a heads up that we might need some extra time to
> finalize things before an RC can be cut.
>
> Thanks,
> Andrew
>
> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> s29752-hadoopdev@yahoo.com.invalid> wrote:
>
> > Hi,
> >
> > I am worry about KMS and transparent encryption since there are quite
> many
> > bugs discovered after it got merged to branch-2.  It gives us an
> impression
> > that the feature is not yet well tested.  Indeed, transparent encryption
> is
> > a complicated feature which changes the core part of HDFS.  It is not
> easy
> > to get everything right.
> >
> >
> > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > feature.  It introduces a new storage type ARCHIVE and the concept of
> block
> > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> storage,
> > the blocks will be stored using the appropriate storage types specified
> by
> > storage policies assigned to the files/directories.  Cluster admin could
> > disable the feature by simply not configuring any storage type and not
> > setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is
> > in the final stages to be merged to branch-2.
> >
> > Regards,
> > Tsz-Wo
> >
> >
> >
> > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > suresh@hortonworks.com> wrote:
> >
> >
> > >
> > >
> > >I actually would like to see both archival storage and single replica
> > >memory writes to be in 2.6 release. Archival storage is in the final
> > stages
> > >of getting ready for branch-2 merge as Nicholas has already indicated on
> > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> these
> > >features are being in development for sometime.
> > >
> > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> > >wrote:
> > >
> > >> Hey Arun,
> > >>
> > >> Maybe we could do a quick run through of the Roadmap wiki and
> > add/retarget
> > >> things accordingly?
> > >>
> > >> I think the KMS and transparent encryption are ready to go. We've got
> a
> > >> very few further bug fixes pending, but that's it.
> > >>
> > >> Two HDFS things that I think probably won't make the end of the week
> are
> > >> archival storage (HDFS-6584) and single replica memory writes
> > (HDFS-6581),
> > >> which I believe are under the HSM banner. HDFS-6484 was just merged to
> > >> trunk and I think needs a little more work before it goes into
> branch-2.
> > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> > off
> > >> yet.
> > >>
> > >> Just my 2c as I did not work directly on these features. I just
> > generally
> > >> shy away from shipping bits quite this fresh.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > wrote:
> > >>
> > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > nicely.
> > >> >
> > >> > I'll create branch-2.6 by end of the week and we can go from there
> to
> > >> > stabilize it - hopefully in the next few weeks.
> > >> >
> > >> > Thoughts?
> > >> >
> > >> > thanks,
> > >> > Arun
> > >> >
> > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <acm@hortonworks.com
> >
> > >> > wrote:
> > >> >
> > >> > > Folks,
> > >> > >
> > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > >> > > hadoop-2.6.
> > >> > >
> > >> > >  Currently, here is the Roadmap per the wiki:
> > >> > >
> > >> > >         • HADOOP
> > >> > >                 • Credential provider HADOOP-10607
> > >> > >         • HDFS
> > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> for
> > >> > using
> > >> > > storage tiers by the applications HDFS-5682
> > >> > >                 • Memory as storage tier HDFS-5851
> > >> > >         • YARN
> > >> > >                 • Dynamic Resource Configuration YARN-291
> > >> > >                 • NodeManager Restart YARN-1336
> > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > >> > >                 • Support for admin-specified labels in YARN
> > YARN-796
> > >> > >                 • Support for automatic, shared cache for YARN
> > >> > application
> > >> > > artifacts YARN-1492
> > >> > >                 • Support NodeGroup layer topology on YARN YARN-18
> > >> > >                 • Support for Docker containers in YARN YARN-1964
> > >> > >                 • YARN service registry YARN-913
> > >> > >
> > >> > >  My suspicion is, as is normal, some will make the cut and some
> > won't.
> > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > would
> > >> be
> > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > >> > cadence.
> > >> > >
> > >> > >  More importantly, as we discussed previously, we'd like
> hadoop-2.6
> > to
> > >> be
> > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> start
> > a
> > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> and
> > >> see
> > >> > > how they feel about this.
> > >> > >
> > >> > > thanks,
> > >> > > Arun
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > --
> > >> > Arun C. Murthy
> > >> > Hortonworks Inc.
> > >> > http://hortonworks.com/
> > >> >
> > >> > --
> > >> > CONFIDENTIALITY NOTICE
> > >> > NOTICE: This message is intended for the use of the individual or
> > entity
> > >> to
> > >> > which it is addressed and may contain information that is
> > confidential,
> > >> > privileged and exempt from disclosure under applicable law. If the
> > reader
> > >> > of this message is not the intended recipient, you are hereby
> notified
> > >> that
> > >> > any printing, copying, dissemination, distribution, disclosure or
> > >> > forwarding of this communication is strictly prohibited. If you have
> > >> > received this communication in error, please contact the sender
> > >> immediately
> > >> > and delete it from your system. Thank You.
> > >> >
> > >>
> > >
> > >
> > >
> > >--
> > >http://hortonworks.com/download/
> > >
> > >--
> > >CONFIDENTIALITY NOTICE
> > >NOTICE: This message is intended for the use of the individual or entity
> > to
> > >which it is addressed and may contain information that is confidential,
> > >privileged and exempt from disclosure under applicable law. If the
> reader
> > >of this message is not the intended recipient, you are hereby notified
> > that
> > >any printing, copying, dissemination, distribution, disclosure or
> > >forwarding of this communication is strictly prohibited. If you have
> > >received this communication in error, please contact the sender
> > immediately
> > >and delete it from your system. Thank You.
> > >
> > >
> >
>



-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Suresh Srinivas <su...@hortonworks.com>.

Given some of the features are in final stages of stabilization,
Arun, we should hold off creating 2.6 branch or building an RC by a week?
All the features in flux are important ones and worth delaying the release
by a week.

On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
wrote:

> Hey Nicholas,
>
> My concern about Archival Storage isn't related to the code quality or the
> size of the feature. I think that you and Jing did good work. My concern is
> that once we ship, we're locked into that set of archival storage APIs, and
> these APIs are not yet finalized. Simply being able to turn off the feature
> does not change the compatibility story.
>
> I'm willing to devote time to help review these JIRAs and kick the tires on
> the APIs, but my point above was that I'm not sure it'd all be done by the
> end of the week. Testing might also reveal additional changes that need to
> be made, which also might not happen by end-of-week.
>
> I guess the question before us is if we're comfortable putting something in
> branch-2.6 and then potentially adding API changes after. I'm okay with
> that as long as we're all aware that this might happen.
>
> Arun, as RM is this cool with you? Again, I like this feature and I'm fine
> with it's inclusion, just a heads up that we might need some extra time to
> finalize things before an RC can be cut.
>
> Thanks,
> Andrew
>
> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
> s29752-hadoopdev@yahoo.com.invalid> wrote:
>
> > Hi,
> >
> > I am worry about KMS and transparent encryption since there are quite
> many
> > bugs discovered after it got merged to branch-2.  It gives us an
> impression
> > that the feature is not yet well tested.  Indeed, transparent encryption
> is
> > a complicated feature which changes the core part of HDFS.  It is not
> easy
> > to get everything right.
> >
> >
> > For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> > feature.  It introduces a new storage type ARCHIVE and the concept of
> block
> > storage policy to HDFS.  When a cluster is configured with ARCHIVE
> storage,
> > the blocks will be stored using the appropriate storage types specified
> by
> > storage policies assigned to the files/directories.  Cluster admin could
> > disable the feature by simply not configuring any storage type and not
> > setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is
> > in the final stages to be merged to branch-2.
> >
> > Regards,
> > Tsz-Wo
> >
> >
> >
> > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> > suresh@hortonworks.com> wrote:
> >
> >
> > >
> > >
> > >I actually would like to see both archival storage and single replica
> > >memory writes to be in 2.6 release. Archival storage is in the final
> > stages
> > >of getting ready for branch-2 merge as Nicholas has already indicated on
> > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> these
> > >features are being in development for sometime.
> > >
> > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> > >wrote:
> > >
> > >> Hey Arun,
> > >>
> > >> Maybe we could do a quick run through of the Roadmap wiki and
> > add/retarget
> > >> things accordingly?
> > >>
> > >> I think the KMS and transparent encryption are ready to go. We've got
> a
> > >> very few further bug fixes pending, but that's it.
> > >>
> > >> Two HDFS things that I think probably won't make the end of the week
> are
> > >> archival storage (HDFS-6584) and single replica memory writes
> > (HDFS-6581),
> > >> which I believe are under the HSM banner. HDFS-6484 was just merged to
> > >> trunk and I think needs a little more work before it goes into
> branch-2.
> > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> > off
> > >> yet.
> > >>
> > >> Just my 2c as I did not work directly on these features. I just
> > generally
> > >> shy away from shipping bits quite this fresh.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > wrote:
> > >>
> > >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> > nicely.
> > >> >
> > >> > I'll create branch-2.6 by end of the week and we can go from there
> to
> > >> > stabilize it - hopefully in the next few weeks.
> > >> >
> > >> > Thoughts?
> > >> >
> > >> > thanks,
> > >> > Arun
> > >> >
> > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <acm@hortonworks.com
> >
> > >> > wrote:
> > >> >
> > >> > > Folks,
> > >> > >
> > >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > >> > > hadoop-2.6.
> > >> > >
> > >> > >  Currently, here is the Roadmap per the wiki:
> > >> > >
> > >> > >         • HADOOP
> > >> > >                 • Credential provider HADOOP-10607
> > >> > >         • HDFS
> > >> > >                 • Heterogeneous storage (Phase 2) - Support APIs
> for
> > >> > using
> > >> > > storage tiers by the applications HDFS-5682
> > >> > >                 • Memory as storage tier HDFS-5851
> > >> > >         • YARN
> > >> > >                 • Dynamic Resource Configuration YARN-291
> > >> > >                 • NodeManager Restart YARN-1336
> > >> > >                 • ResourceManager HA Phase 2 YARN-556
> > >> > >                 • Support for admin-specified labels in YARN
> > YARN-796
> > >> > >                 • Support for automatic, shared cache for YARN
> > >> > application
> > >> > > artifacts YARN-1492
> > >> > >                 • Support NodeGroup layer topology on YARN YARN-18
> > >> > >                 • Support for Docker containers in YARN YARN-1964
> > >> > >                 • YARN service registry YARN-913
> > >> > >
> > >> > >  My suspicion is, as is normal, some will make the cut and some
> > won't.
> > >> > > Please do add/subtract from the list as appropriate. Ideally, it
> > would
> > >> be
> > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > >> > cadence.
> > >> > >
> > >> > >  More importantly, as we discussed previously, we'd like
> hadoop-2.6
> > to
> > >> be
> > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> start
> > a
> > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> and
> > >> see
> > >> > > how they feel about this.
> > >> > >
> > >> > > thanks,
> > >> > > Arun
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > --
> > >> > Arun C. Murthy
> > >> > Hortonworks Inc.
> > >> > http://hortonworks.com/
> > >> >
> > >> > --
> > >> > CONFIDENTIALITY NOTICE
> > >> > NOTICE: This message is intended for the use of the individual or
> > entity
> > >> to
> > >> > which it is addressed and may contain information that is
> > confidential,
> > >> > privileged and exempt from disclosure under applicable law. If the
> > reader
> > >> > of this message is not the intended recipient, you are hereby
> notified
> > >> that
> > >> > any printing, copying, dissemination, distribution, disclosure or
> > >> > forwarding of this communication is strictly prohibited. If you have
> > >> > received this communication in error, please contact the sender
> > >> immediately
> > >> > and delete it from your system. Thank You.
> > >> >
> > >>
> > >
> > >
> > >
> > >--
> > >http://hortonworks.com/download/
> > >
> > >--
> > >CONFIDENTIALITY NOTICE
> > >NOTICE: This message is intended for the use of the individual or entity
> > to
> > >which it is addressed and may contain information that is confidential,
> > >privileged and exempt from disclosure under applicable law. If the
> reader
> > >of this message is not the intended recipient, you are hereby notified
> > that
> > >any printing, copying, dissemination, distribution, disclosure or
> > >forwarding of this communication is strictly prohibited. If you have
> > >received this communication in error, please contact the sender
> > immediately
> > >and delete it from your system. Thank You.
> > >
> > >
> >
>



-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

Hey Nicholas,

My concern about Archival Storage isn't related to the code quality or the
size of the feature. I think that you and Jing did good work. My concern is
that once we ship, we're locked into that set of archival storage APIs, and
these APIs are not yet finalized. Simply being able to turn off the feature
does not change the compatibility story.

I'm willing to devote time to help review these JIRAs and kick the tires on
the APIs, but my point above was that I'm not sure it'd all be done by the
end of the week. Testing might also reveal additional changes that need to
be made, which also might not happen by end-of-week.

I guess the question before us is if we're comfortable putting something in
branch-2.6 and then potentially adding API changes after. I'm okay with
that as long as we're all aware that this might happen.

Arun, as RM is this cool with you? Again, I like this feature and I'm fine
with it's inclusion, just a heads up that we might need some extra time to
finalize things before an RC can be cut.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
s29752-hadoopdev@yahoo.com.invalid> wrote:

> Hi,
>
> I am worry about KMS and transparent encryption since there are quite many
> bugs discovered after it got merged to branch-2.  It gives us an impression
> that the feature is not yet well tested.  Indeed, transparent encryption is
> a complicated feature which changes the core part of HDFS.  It is not easy
> to get everything right.
>
>
> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> feature.  It introduces a new storage type ARCHIVE and the concept of block
> storage policy to HDFS.  When a cluster is configured with ARCHIVE storage,
> the blocks will be stored using the appropriate storage types specified by
> storage policies assigned to the files/directories.  Cluster admin could
> disable the feature by simply not configuring any storage type and not
> setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is
> in the final stages to be merged to branch-2.
>
> Regards,
> Tsz-Wo
>
>
>
> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> suresh@hortonworks.com> wrote:
>
>
> >
> >
> >I actually would like to see both archival storage and single replica
> >memory writes to be in 2.6 release. Archival storage is in the final
> stages
> >of getting ready for branch-2 merge as Nicholas has already indicated on
> >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> >features are being in development for sometime.
> >
> >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> >wrote:
> >
> >> Hey Arun,
> >>
> >> Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> >> things accordingly?
> >>
> >> I think the KMS and transparent encryption are ready to go. We've got a
> >> very few further bug fixes pending, but that's it.
> >>
> >> Two HDFS things that I think probably won't make the end of the week are
> >> archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> >> which I believe are under the HSM banner. HDFS-6484 was just merged to
> >> trunk and I think needs a little more work before it goes into branch-2.
> >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> >> yet.
> >>
> >> Just my 2c as I did not work directly on these features. I just
> generally
> >> shy away from shipping bits quite this fresh.
> >>
> >> Thanks,
> >> Andrew
> >>
> >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >>
> >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> >> >
> >> > I'll create branch-2.6 by end of the week and we can go from there to
> >> > stabilize it - hopefully in the next few weeks.
> >> >
> >> > Thoughts?
> >> >
> >> > thanks,
> >> > Arun
> >> >
> >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> >> > wrote:
> >> >
> >> > > Folks,
> >> > >
> >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> >> > > hadoop-2.6.
> >> > >
> >> > >  Currently, here is the Roadmap per the wiki:
> >> > >
> >> > >         • HADOOP
> >> > >                 • Credential provider HADOOP-10607
> >> > >         • HDFS
> >> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> >> > using
> >> > > storage tiers by the applications HDFS-5682
> >> > >                 • Memory as storage tier HDFS-5851
> >> > >         • YARN
> >> > >                 • Dynamic Resource Configuration YARN-291
> >> > >                 • NodeManager Restart YARN-1336
> >> > >                 • ResourceManager HA Phase 2 YARN-556
> >> > >                 • Support for admin-specified labels in YARN
> YARN-796
> >> > >                 • Support for automatic, shared cache for YARN
> >> > application
> >> > > artifacts YARN-1492
> >> > >                 • Support NodeGroup layer topology on YARN YARN-18
> >> > >                 • Support for Docker containers in YARN YARN-1964
> >> > >                 • YARN service registry YARN-913
> >> > >
> >> > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> >> > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> >> be
> >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> >> > cadence.
> >> > >
> >> > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> >> be
> >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start
> a
> >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> >> see
> >> > > how they feel about this.
> >> > >
> >> > > thanks,
> >> > > Arun
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> >
> >> > --
> >> > Arun C. Murthy
> >> > Hortonworks Inc.
> >> > http://hortonworks.com/
> >> >
> >> > --
> >> > CONFIDENTIALITY NOTICE
> >> > NOTICE: This message is intended for the use of the individual or
> entity
> >> to
> >> > which it is addressed and may contain information that is
> confidential,
> >> > privileged and exempt from disclosure under applicable law. If the
> reader
> >> > of this message is not the intended recipient, you are hereby notified
> >> that
> >> > any printing, copying, dissemination, distribution, disclosure or
> >> > forwarding of this communication is strictly prohibited. If you have
> >> > received this communication in error, please contact the sender
> >> immediately
> >> > and delete it from your system. Thank You.
> >> >
> >>
> >
> >
> >
> >--
> >http://hortonworks.com/download/
> >
> >--
> >CONFIDENTIALITY NOTICE
> >NOTICE: This message is intended for the use of the individual or entity
> to
> >which it is addressed and may contain information that is confidential,
> >privileged and exempt from disclosure under applicable law. If the reader
> >of this message is not the intended recipient, you are hereby notified
> that
> >any printing, copying, dissemination, distribution, disclosure or
> >forwarding of this communication is strictly prohibited. If you have
> >received this communication in error, please contact the sender
> immediately
> >and delete it from your system. Thank You.
> >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

Hey Nicholas,

My concern about Archival Storage isn't related to the code quality or the
size of the feature. I think that you and Jing did good work. My concern is
that once we ship, we're locked into that set of archival storage APIs, and
these APIs are not yet finalized. Simply being able to turn off the feature
does not change the compatibility story.

I'm willing to devote time to help review these JIRAs and kick the tires on
the APIs, but my point above was that I'm not sure it'd all be done by the
end of the week. Testing might also reveal additional changes that need to
be made, which also might not happen by end-of-week.

I guess the question before us is if we're comfortable putting something in
branch-2.6 and then potentially adding API changes after. I'm okay with
that as long as we're all aware that this might happen.

Arun, as RM is this cool with you? Again, I like this feature and I'm fine
with it's inclusion, just a heads up that we might need some extra time to
finalize things before an RC can be cut.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
s29752-hadoopdev@yahoo.com.invalid> wrote:

> Hi,
>
> I am worry about KMS and transparent encryption since there are quite many
> bugs discovered after it got merged to branch-2.  It gives us an impression
> that the feature is not yet well tested.  Indeed, transparent encryption is
> a complicated feature which changes the core part of HDFS.  It is not easy
> to get everything right.
>
>
> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> feature.  It introduces a new storage type ARCHIVE and the concept of block
> storage policy to HDFS.  When a cluster is configured with ARCHIVE storage,
> the blocks will be stored using the appropriate storage types specified by
> storage policies assigned to the files/directories.  Cluster admin could
> disable the feature by simply not configuring any storage type and not
> setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is
> in the final stages to be merged to branch-2.
>
> Regards,
> Tsz-Wo
>
>
>
> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> suresh@hortonworks.com> wrote:
>
>
> >
> >
> >I actually would like to see both archival storage and single replica
> >memory writes to be in 2.6 release. Archival storage is in the final
> stages
> >of getting ready for branch-2 merge as Nicholas has already indicated on
> >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> >features are being in development for sometime.
> >
> >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> >wrote:
> >
> >> Hey Arun,
> >>
> >> Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> >> things accordingly?
> >>
> >> I think the KMS and transparent encryption are ready to go. We've got a
> >> very few further bug fixes pending, but that's it.
> >>
> >> Two HDFS things that I think probably won't make the end of the week are
> >> archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> >> which I believe are under the HSM banner. HDFS-6484 was just merged to
> >> trunk and I think needs a little more work before it goes into branch-2.
> >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> >> yet.
> >>
> >> Just my 2c as I did not work directly on these features. I just
> generally
> >> shy away from shipping bits quite this fresh.
> >>
> >> Thanks,
> >> Andrew
> >>
> >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >>
> >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> >> >
> >> > I'll create branch-2.6 by end of the week and we can go from there to
> >> > stabilize it - hopefully in the next few weeks.
> >> >
> >> > Thoughts?
> >> >
> >> > thanks,
> >> > Arun
> >> >
> >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> >> > wrote:
> >> >
> >> > > Folks,
> >> > >
> >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> >> > > hadoop-2.6.
> >> > >
> >> > >  Currently, here is the Roadmap per the wiki:
> >> > >
> >> > >         • HADOOP
> >> > >                 • Credential provider HADOOP-10607
> >> > >         • HDFS
> >> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> >> > using
> >> > > storage tiers by the applications HDFS-5682
> >> > >                 • Memory as storage tier HDFS-5851
> >> > >         • YARN
> >> > >                 • Dynamic Resource Configuration YARN-291
> >> > >                 • NodeManager Restart YARN-1336
> >> > >                 • ResourceManager HA Phase 2 YARN-556
> >> > >                 • Support for admin-specified labels in YARN
> YARN-796
> >> > >                 • Support for automatic, shared cache for YARN
> >> > application
> >> > > artifacts YARN-1492
> >> > >                 • Support NodeGroup layer topology on YARN YARN-18
> >> > >                 • Support for Docker containers in YARN YARN-1964
> >> > >                 • YARN service registry YARN-913
> >> > >
> >> > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> >> > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> >> be
> >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> >> > cadence.
> >> > >
> >> > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> >> be
> >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start
> a
> >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> >> see
> >> > > how they feel about this.
> >> > >
> >> > > thanks,
> >> > > Arun
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> >
> >> > --
> >> > Arun C. Murthy
> >> > Hortonworks Inc.
> >> > http://hortonworks.com/
> >> >
> >> > --
> >> > CONFIDENTIALITY NOTICE
> >> > NOTICE: This message is intended for the use of the individual or
> entity
> >> to
> >> > which it is addressed and may contain information that is
> confidential,
> >> > privileged and exempt from disclosure under applicable law. If the
> reader
> >> > of this message is not the intended recipient, you are hereby notified
> >> that
> >> > any printing, copying, dissemination, distribution, disclosure or
> >> > forwarding of this communication is strictly prohibited. If you have
> >> > received this communication in error, please contact the sender
> >> immediately
> >> > and delete it from your system. Thank You.
> >> >
> >>
> >
> >
> >
> >--
> >http://hortonworks.com/download/
> >
> >--
> >CONFIDENTIALITY NOTICE
> >NOTICE: This message is intended for the use of the individual or entity
> to
> >which it is addressed and may contain information that is confidential,
> >privileged and exempt from disclosure under applicable law. If the reader
> >of this message is not the intended recipient, you are hereby notified
> that
> >any printing, copying, dissemination, distribution, disclosure or
> >forwarding of this communication is strictly prohibited. If you have
> >received this communication in error, please contact the sender
> immediately
> >and delete it from your system. Thank You.
> >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

Hey Nicholas,

My concern about Archival Storage isn't related to the code quality or the
size of the feature. I think that you and Jing did good work. My concern is
that once we ship, we're locked into that set of archival storage APIs, and
these APIs are not yet finalized. Simply being able to turn off the feature
does not change the compatibility story.

I'm willing to devote time to help review these JIRAs and kick the tires on
the APIs, but my point above was that I'm not sure it'd all be done by the
end of the week. Testing might also reveal additional changes that need to
be made, which also might not happen by end-of-week.

I guess the question before us is if we're comfortable putting something in
branch-2.6 and then potentially adding API changes after. I'm okay with
that as long as we're all aware that this might happen.

Arun, as RM is this cool with you? Again, I like this feature and I'm fine
with it's inclusion, just a heads up that we might need some extra time to
finalize things before an RC can be cut.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
s29752-hadoopdev@yahoo.com.invalid> wrote:

> Hi,
>
> I am worry about KMS and transparent encryption since there are quite many
> bugs discovered after it got merged to branch-2.  It gives us an impression
> that the feature is not yet well tested.  Indeed, transparent encryption is
> a complicated feature which changes the core part of HDFS.  It is not easy
> to get everything right.
>
>
> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> feature.  It introduces a new storage type ARCHIVE and the concept of block
> storage policy to HDFS.  When a cluster is configured with ARCHIVE storage,
> the blocks will be stored using the appropriate storage types specified by
> storage policies assigned to the files/directories.  Cluster admin could
> disable the feature by simply not configuring any storage type and not
> setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is
> in the final stages to be merged to branch-2.
>
> Regards,
> Tsz-Wo
>
>
>
> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> suresh@hortonworks.com> wrote:
>
>
> >
> >
> >I actually would like to see both archival storage and single replica
> >memory writes to be in 2.6 release. Archival storage is in the final
> stages
> >of getting ready for branch-2 merge as Nicholas has already indicated on
> >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> >features are being in development for sometime.
> >
> >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> >wrote:
> >
> >> Hey Arun,
> >>
> >> Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> >> things accordingly?
> >>
> >> I think the KMS and transparent encryption are ready to go. We've got a
> >> very few further bug fixes pending, but that's it.
> >>
> >> Two HDFS things that I think probably won't make the end of the week are
> >> archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> >> which I believe are under the HSM banner. HDFS-6484 was just merged to
> >> trunk and I think needs a little more work before it goes into branch-2.
> >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> >> yet.
> >>
> >> Just my 2c as I did not work directly on these features. I just
> generally
> >> shy away from shipping bits quite this fresh.
> >>
> >> Thanks,
> >> Andrew
> >>
> >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >>
> >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> >> >
> >> > I'll create branch-2.6 by end of the week and we can go from there to
> >> > stabilize it - hopefully in the next few weeks.
> >> >
> >> > Thoughts?
> >> >
> >> > thanks,
> >> > Arun
> >> >
> >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> >> > wrote:
> >> >
> >> > > Folks,
> >> > >
> >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> >> > > hadoop-2.6.
> >> > >
> >> > >  Currently, here is the Roadmap per the wiki:
> >> > >
> >> > >         • HADOOP
> >> > >                 • Credential provider HADOOP-10607
> >> > >         • HDFS
> >> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> >> > using
> >> > > storage tiers by the applications HDFS-5682
> >> > >                 • Memory as storage tier HDFS-5851
> >> > >         • YARN
> >> > >                 • Dynamic Resource Configuration YARN-291
> >> > >                 • NodeManager Restart YARN-1336
> >> > >                 • ResourceManager HA Phase 2 YARN-556
> >> > >                 • Support for admin-specified labels in YARN
> YARN-796
> >> > >                 • Support for automatic, shared cache for YARN
> >> > application
> >> > > artifacts YARN-1492
> >> > >                 • Support NodeGroup layer topology on YARN YARN-18
> >> > >                 • Support for Docker containers in YARN YARN-1964
> >> > >                 • YARN service registry YARN-913
> >> > >
> >> > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> >> > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> >> be
> >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> >> > cadence.
> >> > >
> >> > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> >> be
> >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start
> a
> >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> >> see
> >> > > how they feel about this.
> >> > >
> >> > > thanks,
> >> > > Arun
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> >
> >> > --
> >> > Arun C. Murthy
> >> > Hortonworks Inc.
> >> > http://hortonworks.com/
> >> >
> >> > --
> >> > CONFIDENTIALITY NOTICE
> >> > NOTICE: This message is intended for the use of the individual or
> entity
> >> to
> >> > which it is addressed and may contain information that is
> confidential,
> >> > privileged and exempt from disclosure under applicable law. If the
> reader
> >> > of this message is not the intended recipient, you are hereby notified
> >> that
> >> > any printing, copying, dissemination, distribution, disclosure or
> >> > forwarding of this communication is strictly prohibited. If you have
> >> > received this communication in error, please contact the sender
> >> immediately
> >> > and delete it from your system. Thank You.
> >> >
> >>
> >
> >
> >
> >--
> >http://hortonworks.com/download/
> >
> >--
> >CONFIDENTIALITY NOTICE
> >NOTICE: This message is intended for the use of the individual or entity
> to
> >which it is addressed and may contain information that is confidential,
> >privileged and exempt from disclosure under applicable law. If the reader
> >of this message is not the intended recipient, you are hereby notified
> that
> >any printing, copying, dissemination, distribution, disclosure or
> >forwarding of this communication is strictly prohibited. If you have
> >received this communication in error, please contact the sender
> immediately
> >and delete it from your system. Thank You.
> >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

Hey Nicholas,

My concern about Archival Storage isn't related to the code quality or the
size of the feature. I think that you and Jing did good work. My concern is
that once we ship, we're locked into that set of archival storage APIs, and
these APIs are not yet finalized. Simply being able to turn off the feature
does not change the compatibility story.

I'm willing to devote time to help review these JIRAs and kick the tires on
the APIs, but my point above was that I'm not sure it'd all be done by the
end of the week. Testing might also reveal additional changes that need to
be made, which also might not happen by end-of-week.

I guess the question before us is if we're comfortable putting something in
branch-2.6 and then potentially adding API changes after. I'm okay with
that as long as we're all aware that this might happen.

Arun, as RM is this cool with you? Again, I like this feature and I'm fine
with it's inclusion, just a heads up that we might need some extra time to
finalize things before an RC can be cut.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
s29752-hadoopdev@yahoo.com.invalid> wrote:

> Hi,
>
> I am worry about KMS and transparent encryption since there are quite many
> bugs discovered after it got merged to branch-2.  It gives us an impression
> that the feature is not yet well tested.  Indeed, transparent encryption is
> a complicated feature which changes the core part of HDFS.  It is not easy
> to get everything right.
>
>
> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
> feature.  It introduces a new storage type ARCHIVE and the concept of block
> storage policy to HDFS.  When a cluster is configured with ARCHIVE storage,
> the blocks will be stored using the appropriate storage types specified by
> storage policies assigned to the files/directories.  Cluster admin could
> disable the feature by simply not configuring any storage type and not
> setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is
> in the final stages to be merged to branch-2.
>
> Regards,
> Tsz-Wo
>
>
>
> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
> suresh@hortonworks.com> wrote:
>
>
> >
> >
> >I actually would like to see both archival storage and single replica
> >memory writes to be in 2.6 release. Archival storage is in the final
> stages
> >of getting ready for branch-2 merge as Nicholas has already indicated on
> >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> >features are being in development for sometime.
> >
> >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> >wrote:
> >
> >> Hey Arun,
> >>
> >> Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> >> things accordingly?
> >>
> >> I think the KMS and transparent encryption are ready to go. We've got a
> >> very few further bug fixes pending, but that's it.
> >>
> >> Two HDFS things that I think probably won't make the end of the week are
> >> archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> >> which I believe are under the HSM banner. HDFS-6484 was just merged to
> >> trunk and I think needs a little more work before it goes into branch-2.
> >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> >> yet.
> >>
> >> Just my 2c as I did not work directly on these features. I just
> generally
> >> shy away from shipping bits quite this fresh.
> >>
> >> Thanks,
> >> Andrew
> >>
> >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >>
> >> > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> >> >
> >> > I'll create branch-2.6 by end of the week and we can go from there to
> >> > stabilize it - hopefully in the next few weeks.
> >> >
> >> > Thoughts?
> >> >
> >> > thanks,
> >> > Arun
> >> >
> >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> >> > wrote:
> >> >
> >> > > Folks,
> >> > >
> >> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> >> > > hadoop-2.6.
> >> > >
> >> > >  Currently, here is the Roadmap per the wiki:
> >> > >
> >> > >         • HADOOP
> >> > >                 • Credential provider HADOOP-10607
> >> > >         • HDFS
> >> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> >> > using
> >> > > storage tiers by the applications HDFS-5682
> >> > >                 • Memory as storage tier HDFS-5851
> >> > >         • YARN
> >> > >                 • Dynamic Resource Configuration YARN-291
> >> > >                 • NodeManager Restart YARN-1336
> >> > >                 • ResourceManager HA Phase 2 YARN-556
> >> > >                 • Support for admin-specified labels in YARN
> YARN-796
> >> > >                 • Support for automatic, shared cache for YARN
> >> > application
> >> > > artifacts YARN-1492
> >> > >                 • Support NodeGroup layer topology on YARN YARN-18
> >> > >                 • Support for Docker containers in YARN YARN-1964
> >> > >                 • YARN service registry YARN-913
> >> > >
> >> > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> >> > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> >> be
> >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> >> > cadence.
> >> > >
> >> > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> >> be
> >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start
> a
> >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> >> see
> >> > > how they feel about this.
> >> > >
> >> > > thanks,
> >> > > Arun
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> >
> >> > --
> >> > Arun C. Murthy
> >> > Hortonworks Inc.
> >> > http://hortonworks.com/
> >> >
> >> > --
> >> > CONFIDENTIALITY NOTICE
> >> > NOTICE: This message is intended for the use of the individual or
> entity
> >> to
> >> > which it is addressed and may contain information that is
> confidential,
> >> > privileged and exempt from disclosure under applicable law. If the
> reader
> >> > of this message is not the intended recipient, you are hereby notified
> >> that
> >> > any printing, copying, dissemination, distribution, disclosure or
> >> > forwarding of this communication is strictly prohibited. If you have
> >> > received this communication in error, please contact the sender
> >> immediately
> >> > and delete it from your system. Thank You.
> >> >
> >>
> >
> >
> >
> >--
> >http://hortonworks.com/download/
> >
> >--
> >CONFIDENTIALITY NOTICE
> >NOTICE: This message is intended for the use of the individual or entity
> to
> >which it is addressed and may contain information that is confidential,
> >privileged and exempt from disclosure under applicable law. If the reader
> >of this message is not the intended recipient, you are hereby notified
> that
> >any printing, copying, dissemination, distribution, disclosure or
> >forwarding of this communication is strictly prohibited. If you have
> >received this communication in error, please contact the sender
> immediately
> >and delete it from your system. Thank You.
> >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by "Tsz Wo (Nicholas), Sze" <s2...@yahoo.com.INVALID>.

Hi,

I am worry about KMS and transparent encryption since there are quite many bugs discovered after it got merged to branch-2.  It gives us an impression that the feature is not yet well tested.  Indeed, transparent encryption is a complicated feature which changes the core part of HDFS.  It is not easy to get everything right.


For HDFS-6584: Archival Storage, it is a relatively simple and low risk feature.  It introduces a new storage type ARCHIVE and the concept of block storage policy to HDFS.  When a cluster is configured with ARCHIVE storage, the blocks will be stored using the appropriate storage types specified by storage policies assigned to the files/directories.  Cluster admin could disable the feature by simply not configuring any storage type and not setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is in the final stages to be merged to branch-2.

Regards,
Tsz-Wo



On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <su...@hortonworks.com> wrote:
 

>
>
>I actually would like to see both archival storage and single replica
>memory writes to be in 2.6 release. Archival storage is in the final stages
>of getting ready for branch-2 merge as Nicholas has already indicated on
>the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
>features are being in development for sometime.
>
>On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
>wrote:
>
>> Hey Arun,
>>
>> Maybe we could do a quick run through of the Roadmap wiki and add/retarget
>> things accordingly?
>>
>> I think the KMS and transparent encryption are ready to go. We've got a
>> very few further bug fixes pending, but that's it.
>>
>> Two HDFS things that I think probably won't make the end of the week are
>> archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
>> which I believe are under the HSM banner. HDFS-6484 was just merged to
>> trunk and I think needs a little more work before it goes into branch-2.
>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
>> yet.
>>
>> Just my 2c as I did not work directly on these features. I just generally
>> shy away from shipping bits quite this fresh.
>>
>> Thanks,
>> Andrew
>>
>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:
>>
>> > Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>> >
>> > I'll create branch-2.6 by end of the week and we can go from there to
>> > stabilize it - hopefully in the next few weeks.
>> >
>> > Thoughts?
>> >
>> > thanks,
>> > Arun
>> >
>> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
>> > wrote:
>> >
>> > > Folks,
>> > >
>> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
>> > > hadoop-2.6.
>> > >
>> > >  Currently, here is the Roadmap per the wiki:
>> > >
>> > >         • HADOOP
>> > >                 • Credential provider HADOOP-10607
>> > >         • HDFS
>> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
>> > using
>> > > storage tiers by the applications HDFS-5682
>> > >                 • Memory as storage tier HDFS-5851
>> > >         • YARN
>> > >                 • Dynamic Resource Configuration YARN-291
>> > >                 • NodeManager Restart YARN-1336
>> > >                 • ResourceManager HA Phase 2 YARN-556
>> > >                 • Support for admin-specified labels in YARN YARN-796
>> > >                 • Support for automatic, shared cache for YARN
>> > application
>> > > artifacts YARN-1492
>> > >                 • Support NodeGroup layer topology on YARN YARN-18
>> > >                 • Support for Docker containers in YARN YARN-1964
>> > >                 • YARN service registry YARN-913
>> > >
>> > >  My suspicion is, as is normal, some will make the cut and some won't.
>> > > Please do add/subtract from the list as appropriate. Ideally, it would
>> be
>> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
>> > cadence.
>> > >
>> > >  More importantly, as we discussed previously, we'd like hadoop-2.6 to
>> be
>> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
>> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
>> see
>> > > how they feel about this.
>> > >
>> > > thanks,
>> > > Arun
>> > >
>> > >
>> >
>> >
>> > --
>> >
>> > --
>> > Arun C. Murthy
>> > Hortonworks Inc.
>> > http://hortonworks.com/
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>
>
>-- 
>http://hortonworks.com/download/
>
>-- 
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity to 
>which it is addressed and may contain information that is confidential, 
>privileged and exempt from disclosure under applicable law. If the reader 
>of this message is not the intended recipient, you are hereby notified that 
>any printing, copying, dissemination, distribution, disclosure or 
>forwarding of this communication is strictly prohibited. If you have 
>received this communication in error, please contact the sender immediately 
>and delete it from your system. Thank You.
>
>

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

I think we should delay cutting the branch beyond the end of the week then,
seeing as the HDFS-6581 merge vote (assuming it goes through) doesn't close
until 12am on Tuesday next week. Archival storage is also still finishing
up user APIs and I had a few other q's posted on HDFS-7081, so again might
not make it by the end of the week.

I don't think it'd be too bad to retarget these to 2.7 though, since Arun
said he plans to roll 2.7 shortly after 2.6. I think a few more weeks would
really seal the deal.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> I actually would like to see both archival storage and single replica
> memory writes to be in 2.6 release. Archival storage is in the final stages
> of getting ready for branch-2 merge as Nicholas has already indicated on
> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> features are being in development for sometime.
>
> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Arun,
> >
> > Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> > things accordingly?
> >
> > I think the KMS and transparent encryption are ready to go. We've got a
> > very few further bug fixes pending, but that's it.
> >
> > Two HDFS things that I think probably won't make the end of the week are
> > archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > trunk and I think needs a little more work before it goes into branch-2.
> > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> > yet.
> >
> > Just my 2c as I did not work directly on these features. I just generally
> > shy away from shipping bits quite this fresh.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >
> > > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> > >
> > > I'll create branch-2.6 by end of the week and we can go from there to
> > > stabilize it - hopefully in the next few weeks.
> > >
> > > Thoughts?
> > >
> > > thanks,
> > > Arun
> > >
> > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > hadoop-2.6.
> > > >
> > > >  Currently, here is the Roadmap per the wiki:
> > > >
> > > >         • HADOOP
> > > >                 • Credential provider HADOOP-10607
> > > >         • HDFS
> > > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > > using
> > > > storage tiers by the applications HDFS-5682
> > > >                 • Memory as storage tier HDFS-5851
> > > >         • YARN
> > > >                 • Dynamic Resource Configuration YARN-291
> > > >                 • NodeManager Restart YARN-1336
> > > >                 • ResourceManager HA Phase 2 YARN-556
> > > >                 • Support for admin-specified labels in YARN YARN-796
> > > >                 • Support for automatic, shared cache for YARN
> > > application
> > > > artifacts YARN-1492
> > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > >                 • Support for Docker containers in YARN YARN-1964
> > > >                 • YARN service registry YARN-913
> > > >
> > > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> > > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> > be
> > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > cadence.
> > > >
> > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> > be
> > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> > see
> > > > how they feel about this.
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

I think we should delay cutting the branch beyond the end of the week then,
seeing as the HDFS-6581 merge vote (assuming it goes through) doesn't close
until 12am on Tuesday next week. Archival storage is also still finishing
up user APIs and I had a few other q's posted on HDFS-7081, so again might
not make it by the end of the week.

I don't think it'd be too bad to retarget these to 2.7 though, since Arun
said he plans to roll 2.7 shortly after 2.6. I think a few more weeks would
really seal the deal.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> I actually would like to see both archival storage and single replica
> memory writes to be in 2.6 release. Archival storage is in the final stages
> of getting ready for branch-2 merge as Nicholas has already indicated on
> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> features are being in development for sometime.
>
> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Arun,
> >
> > Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> > things accordingly?
> >
> > I think the KMS and transparent encryption are ready to go. We've got a
> > very few further bug fixes pending, but that's it.
> >
> > Two HDFS things that I think probably won't make the end of the week are
> > archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > trunk and I think needs a little more work before it goes into branch-2.
> > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> > yet.
> >
> > Just my 2c as I did not work directly on these features. I just generally
> > shy away from shipping bits quite this fresh.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >
> > > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> > >
> > > I'll create branch-2.6 by end of the week and we can go from there to
> > > stabilize it - hopefully in the next few weeks.
> > >
> > > Thoughts?
> > >
> > > thanks,
> > > Arun
> > >
> > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > hadoop-2.6.
> > > >
> > > >  Currently, here is the Roadmap per the wiki:
> > > >
> > > >         • HADOOP
> > > >                 • Credential provider HADOOP-10607
> > > >         • HDFS
> > > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > > using
> > > > storage tiers by the applications HDFS-5682
> > > >                 • Memory as storage tier HDFS-5851
> > > >         • YARN
> > > >                 • Dynamic Resource Configuration YARN-291
> > > >                 • NodeManager Restart YARN-1336
> > > >                 • ResourceManager HA Phase 2 YARN-556
> > > >                 • Support for admin-specified labels in YARN YARN-796
> > > >                 • Support for automatic, shared cache for YARN
> > > application
> > > > artifacts YARN-1492
> > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > >                 • Support for Docker containers in YARN YARN-1964
> > > >                 • YARN service registry YARN-913
> > > >
> > > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> > > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> > be
> > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > cadence.
> > > >
> > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> > be
> > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> > see
> > > > how they feel about this.
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by "Tsz Wo (Nicholas), Sze" <s2...@yahoo.com.INVALID>.

Hi,

I am worry about KMS and transparent encryption since there are quite many bugs discovered after it got merged to branch-2.  It gives us an impression that the feature is not yet well tested.  Indeed, transparent encryption is a complicated feature which changes the core part of HDFS.  It is not easy to get everything right.


For HDFS-6584: Archival Storage, it is a relatively simple and low risk feature.  It introduces a new storage type ARCHIVE and the concept of block storage policy to HDFS.  When a cluster is configured with ARCHIVE storage, the blocks will be stored using the appropriate storage types specified by storage policies assigned to the files/directories.  Cluster admin could disable the feature by simply not configuring any storage type and not setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is in the final stages to be merged to branch-2.

Regards,
Tsz-Wo



On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <su...@hortonworks.com> wrote:
 

>
>
>I actually would like to see both archival storage and single replica
>memory writes to be in 2.6 release. Archival storage is in the final stages
>of getting ready for branch-2 merge as Nicholas has already indicated on
>the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
>features are being in development for sometime.
>
>On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
>wrote:
>
>> Hey Arun,
>>
>> Maybe we could do a quick run through of the Roadmap wiki and add/retarget
>> things accordingly?
>>
>> I think the KMS and transparent encryption are ready to go. We've got a
>> very few further bug fixes pending, but that's it.
>>
>> Two HDFS things that I think probably won't make the end of the week are
>> archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
>> which I believe are under the HSM banner. HDFS-6484 was just merged to
>> trunk and I think needs a little more work before it goes into branch-2.
>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
>> yet.
>>
>> Just my 2c as I did not work directly on these features. I just generally
>> shy away from shipping bits quite this fresh.
>>
>> Thanks,
>> Andrew
>>
>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:
>>
>> > Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>> >
>> > I'll create branch-2.6 by end of the week and we can go from there to
>> > stabilize it - hopefully in the next few weeks.
>> >
>> > Thoughts?
>> >
>> > thanks,
>> > Arun
>> >
>> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
>> > wrote:
>> >
>> > > Folks,
>> > >
>> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
>> > > hadoop-2.6.
>> > >
>> > >  Currently, here is the Roadmap per the wiki:
>> > >
>> > >         • HADOOP
>> > >                 • Credential provider HADOOP-10607
>> > >         • HDFS
>> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
>> > using
>> > > storage tiers by the applications HDFS-5682
>> > >                 • Memory as storage tier HDFS-5851
>> > >         • YARN
>> > >                 • Dynamic Resource Configuration YARN-291
>> > >                 • NodeManager Restart YARN-1336
>> > >                 • ResourceManager HA Phase 2 YARN-556
>> > >                 • Support for admin-specified labels in YARN YARN-796
>> > >                 • Support for automatic, shared cache for YARN
>> > application
>> > > artifacts YARN-1492
>> > >                 • Support NodeGroup layer topology on YARN YARN-18
>> > >                 • Support for Docker containers in YARN YARN-1964
>> > >                 • YARN service registry YARN-913
>> > >
>> > >  My suspicion is, as is normal, some will make the cut and some won't.
>> > > Please do add/subtract from the list as appropriate. Ideally, it would
>> be
>> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
>> > cadence.
>> > >
>> > >  More importantly, as we discussed previously, we'd like hadoop-2.6 to
>> be
>> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
>> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
>> see
>> > > how they feel about this.
>> > >
>> > > thanks,
>> > > Arun
>> > >
>> > >
>> >
>> >
>> > --
>> >
>> > --
>> > Arun C. Murthy
>> > Hortonworks Inc.
>> > http://hortonworks.com/
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>
>
>-- 
>http://hortonworks.com/download/
>
>-- 
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity to 
>which it is addressed and may contain information that is confidential, 
>privileged and exempt from disclosure under applicable law. If the reader 
>of this message is not the intended recipient, you are hereby notified that 
>any printing, copying, dissemination, distribution, disclosure or 
>forwarding of this communication is strictly prohibited. If you have 
>received this communication in error, please contact the sender immediately 
>and delete it from your system. Thank You.
>
>

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

I think we should delay cutting the branch beyond the end of the week then,
seeing as the HDFS-6581 merge vote (assuming it goes through) doesn't close
until 12am on Tuesday next week. Archival storage is also still finishing
up user APIs and I had a few other q's posted on HDFS-7081, so again might
not make it by the end of the week.

I don't think it'd be too bad to retarget these to 2.7 though, since Arun
said he plans to roll 2.7 shortly after 2.6. I think a few more weeks would
really seal the deal.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> I actually would like to see both archival storage and single replica
> memory writes to be in 2.6 release. Archival storage is in the final stages
> of getting ready for branch-2 merge as Nicholas has already indicated on
> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> features are being in development for sometime.
>
> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Arun,
> >
> > Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> > things accordingly?
> >
> > I think the KMS and transparent encryption are ready to go. We've got a
> > very few further bug fixes pending, but that's it.
> >
> > Two HDFS things that I think probably won't make the end of the week are
> > archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > trunk and I think needs a little more work before it goes into branch-2.
> > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> > yet.
> >
> > Just my 2c as I did not work directly on these features. I just generally
> > shy away from shipping bits quite this fresh.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >
> > > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> > >
> > > I'll create branch-2.6 by end of the week and we can go from there to
> > > stabilize it - hopefully in the next few weeks.
> > >
> > > Thoughts?
> > >
> > > thanks,
> > > Arun
> > >
> > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > hadoop-2.6.
> > > >
> > > >  Currently, here is the Roadmap per the wiki:
> > > >
> > > >         • HADOOP
> > > >                 • Credential provider HADOOP-10607
> > > >         • HDFS
> > > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > > using
> > > > storage tiers by the applications HDFS-5682
> > > >                 • Memory as storage tier HDFS-5851
> > > >         • YARN
> > > >                 • Dynamic Resource Configuration YARN-291
> > > >                 • NodeManager Restart YARN-1336
> > > >                 • ResourceManager HA Phase 2 YARN-556
> > > >                 • Support for admin-specified labels in YARN YARN-796
> > > >                 • Support for automatic, shared cache for YARN
> > > application
> > > > artifacts YARN-1492
> > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > >                 • Support for Docker containers in YARN YARN-1964
> > > >                 • YARN service registry YARN-913
> > > >
> > > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> > > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> > be
> > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > cadence.
> > > >
> > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> > be
> > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> > see
> > > > how they feel about this.
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Sangjin Lee <sj...@gmail.com>.

I am also +1 for the end of next week. YARN-1492 (shared cache) is near the
final stages of the review, and with a little more time it would make it
easier to put it in.

On Tue, Sep 23, 2014 at 4:48 PM, Subramaniam V K <su...@gmail.com> wrote:

> +1 for end of next week.
>
> We have got all the patches for YARN-1051 committed to the branch. We are
> currently fixing test-patch & plan to call a merge vote soon. Hopefully it
> will go through by end of next week.
>
> Thanks,
> Subru
>
> On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > Looking at the patches, we might be able to get most of YARN-1492 in by
> the
> > end of next week. There would be a couple of security items still
> > remaining, but we can may be call the feature alpha-ready without them.
> >
> > On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:
> >
> > > I would like to see the shared cache (YARN-1492) make it in as well. We
> > are
> > > going through the final review process (with Karthik and Vinod) and
> > should
> > > be fairly close to complete. This is another feature that has been
> under
> > > development for a while.
> > >
> > > Thanks,
> > > Chris
> > >
> > > On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <
> suresh@hortonworks.com
> > >
> > > wrote:
> > >
> > > > I actually would like to see both archival storage and single replica
> > > > memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > > of getting ready for branch-2 merge as Nicholas has already indicated
> > on
> > > > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > > these
> > > > features are being in development for sometime.
> > > >
> > > > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Hey Arun,
> > > > >
> > > > > Maybe we could do a quick run through of the Roadmap wiki and
> > > > add/retarget
> > > > > things accordingly?
> > > > >
> > > > > I think the KMS and transparent encryption are ready to go. We've
> > got a
> > > > > very few further bug fixes pending, but that's it.
> > > > >
> > > > > Two HDFS things that I think probably won't make the end of the
> week
> > > are
> > > > > archival storage (HDFS-6584) and single replica memory writes
> > > > (HDFS-6581),
> > > > > which I believe are under the HSM banner. HDFS-6484 was just merged
> > to
> > > > > trunk and I think needs a little more work before it goes into
> > > branch-2.
> > > > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> > further
> > > > off
> > > > > yet.
> > > > >
> > > > > Just my 2c as I did not work directly on these features. I just
> > > generally
> > > > > shy away from shipping bits quite this fresh.
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > > > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > > wrote:
> > > > >
> > > > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > > nicely.
> > > > > >
> > > > > > I'll create branch-2.6 by end of the week and we can go from
> there
> > to
> > > > > > stabilize it - hopefully in the next few weeks.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > thanks,
> > > > > > Arun
> > > > > >
> > > > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> > acm@hortonworks.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> > to
> > > > > > > hadoop-2.6.
> > > > > > >
> > > > > > >  Currently, here is the Roadmap per the wiki:
> > > > > > >
> > > > > > >         * HADOOP
> > > > > > >                 * Credential provider HADOOP-10607
> > > > > > >         * HDFS
> > > > > > >                 * Heterogeneous storage (Phase 2) - Support
> APIs
> > > for
> > > > > > using
> > > > > > > storage tiers by the applications HDFS-5682
> > > > > > >                 * Memory as storage tier HDFS-5851
> > > > > > >         * YARN
> > > > > > >                 * Dynamic Resource Configuration YARN-291
> > > > > > >                 * NodeManager Restart YARN-1336
> > > > > > >                 * ResourceManager HA Phase 2 YARN-556
> > > > > > >                 * Support for admin-specified labels in YARN
> > > YARN-796
> > > > > > >                 * Support for automatic, shared cache for YARN
> > > > > > application
> > > > > > > artifacts YARN-1492
> > > > > > >                 * Support NodeGroup layer topology on YARN
> > YARN-18
> > > > > > >                 * Support for Docker containers in YARN
> YARN-1964
> > > > > > >                 * YARN service registry YARN-913
> > > > > > >
> > > > > > >  My suspicion is, as is normal, some will make the cut and some
> > > > won't.
> > > > > > > Please do add/subtract from the list as appropriate. Ideally,
> it
> > > > would
> > > > > be
> > > > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up
> > a
> > > > > > cadence.
> > > > > > >
> > > > > > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > > to
> > > > > be
> > > > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > > start a
> > > > > > > discussion with other communities (HBase, Pig, Hive, Oozie
> etc.)
> > > and
> > > > > see
> > > > > > > how they feel about this.
> > > > > > >
> > > > > > > thanks,
> > > > > > > Arun
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > --
> > > > > > Arun C. Murthy
> > > > > > Hortonworks Inc.
> > > > > > http://hortonworks.com/
> > > > > >
> > > > > > --
> > > > > > CONFIDENTIALITY NOTICE
> > > > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > > > to
> > > > > > which it is addressed and may contain information that is
> > > confidential,
> > > > > > privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > > > of this message is not the intended recipient, you are hereby
> > > notified
> > > > > that
> > > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > > forwarding of this communication is strictly prohibited. If you
> > have
> > > > > > received this communication in error, please contact the sender
> > > > > immediately
> > > > > > and delete it from your system. Thank You.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > http://hortonworks.com/download/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Sangjin Lee <sj...@gmail.com>.

I am also +1 for the end of next week. YARN-1492 (shared cache) is near the
final stages of the review, and with a little more time it would make it
easier to put it in.

On Tue, Sep 23, 2014 at 4:48 PM, Subramaniam V K <su...@gmail.com> wrote:

> +1 for end of next week.
>
> We have got all the patches for YARN-1051 committed to the branch. We are
> currently fixing test-patch & plan to call a merge vote soon. Hopefully it
> will go through by end of next week.
>
> Thanks,
> Subru
>
> On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > Looking at the patches, we might be able to get most of YARN-1492 in by
> the
> > end of next week. There would be a couple of security items still
> > remaining, but we can may be call the feature alpha-ready without them.
> >
> > On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:
> >
> > > I would like to see the shared cache (YARN-1492) make it in as well. We
> > are
> > > going through the final review process (with Karthik and Vinod) and
> > should
> > > be fairly close to complete. This is another feature that has been
> under
> > > development for a while.
> > >
> > > Thanks,
> > > Chris
> > >
> > > On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <
> suresh@hortonworks.com
> > >
> > > wrote:
> > >
> > > > I actually would like to see both archival storage and single replica
> > > > memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > > of getting ready for branch-2 merge as Nicholas has already indicated
> > on
> > > > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > > these
> > > > features are being in development for sometime.
> > > >
> > > > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Hey Arun,
> > > > >
> > > > > Maybe we could do a quick run through of the Roadmap wiki and
> > > > add/retarget
> > > > > things accordingly?
> > > > >
> > > > > I think the KMS and transparent encryption are ready to go. We've
> > got a
> > > > > very few further bug fixes pending, but that's it.
> > > > >
> > > > > Two HDFS things that I think probably won't make the end of the
> week
> > > are
> > > > > archival storage (HDFS-6584) and single replica memory writes
> > > > (HDFS-6581),
> > > > > which I believe are under the HSM banner. HDFS-6484 was just merged
> > to
> > > > > trunk and I think needs a little more work before it goes into
> > > branch-2.
> > > > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> > further
> > > > off
> > > > > yet.
> > > > >
> > > > > Just my 2c as I did not work directly on these features. I just
> > > generally
> > > > > shy away from shipping bits quite this fresh.
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > > > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > > wrote:
> > > > >
> > > > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > > nicely.
> > > > > >
> > > > > > I'll create branch-2.6 by end of the week and we can go from
> there
> > to
> > > > > > stabilize it - hopefully in the next few weeks.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > thanks,
> > > > > > Arun
> > > > > >
> > > > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> > acm@hortonworks.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> > to
> > > > > > > hadoop-2.6.
> > > > > > >
> > > > > > >  Currently, here is the Roadmap per the wiki:
> > > > > > >
> > > > > > >         * HADOOP
> > > > > > >                 * Credential provider HADOOP-10607
> > > > > > >         * HDFS
> > > > > > >                 * Heterogeneous storage (Phase 2) - Support
> APIs
> > > for
> > > > > > using
> > > > > > > storage tiers by the applications HDFS-5682
> > > > > > >                 * Memory as storage tier HDFS-5851
> > > > > > >         * YARN
> > > > > > >                 * Dynamic Resource Configuration YARN-291
> > > > > > >                 * NodeManager Restart YARN-1336
> > > > > > >                 * ResourceManager HA Phase 2 YARN-556
> > > > > > >                 * Support for admin-specified labels in YARN
> > > YARN-796
> > > > > > >                 * Support for automatic, shared cache for YARN
> > > > > > application
> > > > > > > artifacts YARN-1492
> > > > > > >                 * Support NodeGroup layer topology on YARN
> > YARN-18
> > > > > > >                 * Support for Docker containers in YARN
> YARN-1964
> > > > > > >                 * YARN service registry YARN-913
> > > > > > >
> > > > > > >  My suspicion is, as is normal, some will make the cut and some
> > > > won't.
> > > > > > > Please do add/subtract from the list as appropriate. Ideally,
> it
> > > > would
> > > > > be
> > > > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up
> > a
> > > > > > cadence.
> > > > > > >
> > > > > > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > > to
> > > > > be
> > > > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > > start a
> > > > > > > discussion with other communities (HBase, Pig, Hive, Oozie
> etc.)
> > > and
> > > > > see
> > > > > > > how they feel about this.
> > > > > > >
> > > > > > > thanks,
> > > > > > > Arun
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > --
> > > > > > Arun C. Murthy
> > > > > > Hortonworks Inc.
> > > > > > http://hortonworks.com/
> > > > > >
> > > > > > --
> > > > > > CONFIDENTIALITY NOTICE
> > > > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > > > to
> > > > > > which it is addressed and may contain information that is
> > > confidential,
> > > > > > privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > > > of this message is not the intended recipient, you are hereby
> > > notified
> > > > > that
> > > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > > forwarding of this communication is strictly prohibited. If you
> > have
> > > > > > received this communication in error, please contact the sender
> > > > > immediately
> > > > > > and delete it from your system. Thank You.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > http://hortonworks.com/download/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Sangjin Lee <sj...@gmail.com>.

I am also +1 for the end of next week. YARN-1492 (shared cache) is near the
final stages of the review, and with a little more time it would make it
easier to put it in.

On Tue, Sep 23, 2014 at 4:48 PM, Subramaniam V K <su...@gmail.com> wrote:

> +1 for end of next week.
>
> We have got all the patches for YARN-1051 committed to the branch. We are
> currently fixing test-patch & plan to call a merge vote soon. Hopefully it
> will go through by end of next week.
>
> Thanks,
> Subru
>
> On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > Looking at the patches, we might be able to get most of YARN-1492 in by
> the
> > end of next week. There would be a couple of security items still
> > remaining, but we can may be call the feature alpha-ready without them.
> >
> > On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:
> >
> > > I would like to see the shared cache (YARN-1492) make it in as well. We
> > are
> > > going through the final review process (with Karthik and Vinod) and
> > should
> > > be fairly close to complete. This is another feature that has been
> under
> > > development for a while.
> > >
> > > Thanks,
> > > Chris
> > >
> > > On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <
> suresh@hortonworks.com
> > >
> > > wrote:
> > >
> > > > I actually would like to see both archival storage and single replica
> > > > memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > > of getting ready for branch-2 merge as Nicholas has already indicated
> > on
> > > > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > > these
> > > > features are being in development for sometime.
> > > >
> > > > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Hey Arun,
> > > > >
> > > > > Maybe we could do a quick run through of the Roadmap wiki and
> > > > add/retarget
> > > > > things accordingly?
> > > > >
> > > > > I think the KMS and transparent encryption are ready to go. We've
> > got a
> > > > > very few further bug fixes pending, but that's it.
> > > > >
> > > > > Two HDFS things that I think probably won't make the end of the
> week
> > > are
> > > > > archival storage (HDFS-6584) and single replica memory writes
> > > > (HDFS-6581),
> > > > > which I believe are under the HSM banner. HDFS-6484 was just merged
> > to
> > > > > trunk and I think needs a little more work before it goes into
> > > branch-2.
> > > > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> > further
> > > > off
> > > > > yet.
> > > > >
> > > > > Just my 2c as I did not work directly on these features. I just
> > > generally
> > > > > shy away from shipping bits quite this fresh.
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > > > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > > wrote:
> > > > >
> > > > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > > nicely.
> > > > > >
> > > > > > I'll create branch-2.6 by end of the week and we can go from
> there
> > to
> > > > > > stabilize it - hopefully in the next few weeks.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > thanks,
> > > > > > Arun
> > > > > >
> > > > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> > acm@hortonworks.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> > to
> > > > > > > hadoop-2.6.
> > > > > > >
> > > > > > >  Currently, here is the Roadmap per the wiki:
> > > > > > >
> > > > > > >         * HADOOP
> > > > > > >                 * Credential provider HADOOP-10607
> > > > > > >         * HDFS
> > > > > > >                 * Heterogeneous storage (Phase 2) - Support
> APIs
> > > for
> > > > > > using
> > > > > > > storage tiers by the applications HDFS-5682
> > > > > > >                 * Memory as storage tier HDFS-5851
> > > > > > >         * YARN
> > > > > > >                 * Dynamic Resource Configuration YARN-291
> > > > > > >                 * NodeManager Restart YARN-1336
> > > > > > >                 * ResourceManager HA Phase 2 YARN-556
> > > > > > >                 * Support for admin-specified labels in YARN
> > > YARN-796
> > > > > > >                 * Support for automatic, shared cache for YARN
> > > > > > application
> > > > > > > artifacts YARN-1492
> > > > > > >                 * Support NodeGroup layer topology on YARN
> > YARN-18
> > > > > > >                 * Support for Docker containers in YARN
> YARN-1964
> > > > > > >                 * YARN service registry YARN-913
> > > > > > >
> > > > > > >  My suspicion is, as is normal, some will make the cut and some
> > > > won't.
> > > > > > > Please do add/subtract from the list as appropriate. Ideally,
> it
> > > > would
> > > > > be
> > > > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up
> > a
> > > > > > cadence.
> > > > > > >
> > > > > > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > > to
> > > > > be
> > > > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > > start a
> > > > > > > discussion with other communities (HBase, Pig, Hive, Oozie
> etc.)
> > > and
> > > > > see
> > > > > > > how they feel about this.
> > > > > > >
> > > > > > > thanks,
> > > > > > > Arun
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > --
> > > > > > Arun C. Murthy
> > > > > > Hortonworks Inc.
> > > > > > http://hortonworks.com/
> > > > > >
> > > > > > --
> > > > > > CONFIDENTIALITY NOTICE
> > > > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > > > to
> > > > > > which it is addressed and may contain information that is
> > > confidential,
> > > > > > privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > > > of this message is not the intended recipient, you are hereby
> > > notified
> > > > > that
> > > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > > forwarding of this communication is strictly prohibited. If you
> > have
> > > > > > received this communication in error, please contact the sender
> > > > > immediately
> > > > > > and delete it from your system. Thank You.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > http://hortonworks.com/download/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Sangjin Lee <sj...@gmail.com>.

I am also +1 for the end of next week. YARN-1492 (shared cache) is near the
final stages of the review, and with a little more time it would make it
easier to put it in.

On Tue, Sep 23, 2014 at 4:48 PM, Subramaniam V K <su...@gmail.com> wrote:

> +1 for end of next week.
>
> We have got all the patches for YARN-1051 committed to the branch. We are
> currently fixing test-patch & plan to call a merge vote soon. Hopefully it
> will go through by end of next week.
>
> Thanks,
> Subru
>
> On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > Looking at the patches, we might be able to get most of YARN-1492 in by
> the
> > end of next week. There would be a couple of security items still
> > remaining, but we can may be call the feature alpha-ready without them.
> >
> > On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:
> >
> > > I would like to see the shared cache (YARN-1492) make it in as well. We
> > are
> > > going through the final review process (with Karthik and Vinod) and
> > should
> > > be fairly close to complete. This is another feature that has been
> under
> > > development for a while.
> > >
> > > Thanks,
> > > Chris
> > >
> > > On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <
> suresh@hortonworks.com
> > >
> > > wrote:
> > >
> > > > I actually would like to see both archival storage and single replica
> > > > memory writes to be in 2.6 release. Archival storage is in the final
> > > stages
> > > > of getting ready for branch-2 merge as Nicholas has already indicated
> > on
> > > > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > > these
> > > > features are being in development for sometime.
> > > >
> > > > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
> andrew.wang@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Hey Arun,
> > > > >
> > > > > Maybe we could do a quick run through of the Roadmap wiki and
> > > > add/retarget
> > > > > things accordingly?
> > > > >
> > > > > I think the KMS and transparent encryption are ready to go. We've
> > got a
> > > > > very few further bug fixes pending, but that's it.
> > > > >
> > > > > Two HDFS things that I think probably won't make the end of the
> week
> > > are
> > > > > archival storage (HDFS-6584) and single replica memory writes
> > > > (HDFS-6581),
> > > > > which I believe are under the HSM banner. HDFS-6484 was just merged
> > to
> > > > > trunk and I think needs a little more work before it goes into
> > > branch-2.
> > > > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> > further
> > > > off
> > > > > yet.
> > > > >
> > > > > Just my 2c as I did not work directly on these features. I just
> > > generally
> > > > > shy away from shipping bits quite this fresh.
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > > > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > > wrote:
> > > > >
> > > > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > > nicely.
> > > > > >
> > > > > > I'll create branch-2.6 by end of the week and we can go from
> there
> > to
> > > > > > stabilize it - hopefully in the next few weeks.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > thanks,
> > > > > > Arun
> > > > > >
> > > > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> > acm@hortonworks.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> > to
> > > > > > > hadoop-2.6.
> > > > > > >
> > > > > > >  Currently, here is the Roadmap per the wiki:
> > > > > > >
> > > > > > >         * HADOOP
> > > > > > >                 * Credential provider HADOOP-10607
> > > > > > >         * HDFS
> > > > > > >                 * Heterogeneous storage (Phase 2) - Support
> APIs
> > > for
> > > > > > using
> > > > > > > storage tiers by the applications HDFS-5682
> > > > > > >                 * Memory as storage tier HDFS-5851
> > > > > > >         * YARN
> > > > > > >                 * Dynamic Resource Configuration YARN-291
> > > > > > >                 * NodeManager Restart YARN-1336
> > > > > > >                 * ResourceManager HA Phase 2 YARN-556
> > > > > > >                 * Support for admin-specified labels in YARN
> > > YARN-796
> > > > > > >                 * Support for automatic, shared cache for YARN
> > > > > > application
> > > > > > > artifacts YARN-1492
> > > > > > >                 * Support NodeGroup layer topology on YARN
> > YARN-18
> > > > > > >                 * Support for Docker containers in YARN
> YARN-1964
> > > > > > >                 * YARN service registry YARN-913
> > > > > > >
> > > > > > >  My suspicion is, as is normal, some will make the cut and some
> > > > won't.
> > > > > > > Please do add/subtract from the list as appropriate. Ideally,
> it
> > > > would
> > > > > be
> > > > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
> up
> > a
> > > > > > cadence.
> > > > > > >
> > > > > > >  More importantly, as we discussed previously, we'd like
> > hadoop-2.6
> > > > to
> > > > > be
> > > > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > > start a
> > > > > > > discussion with other communities (HBase, Pig, Hive, Oozie
> etc.)
> > > and
> > > > > see
> > > > > > > how they feel about this.
> > > > > > >
> > > > > > > thanks,
> > > > > > > Arun
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > --
> > > > > > Arun C. Murthy
> > > > > > Hortonworks Inc.
> > > > > > http://hortonworks.com/
> > > > > >
> > > > > > --
> > > > > > CONFIDENTIALITY NOTICE
> > > > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > > > to
> > > > > > which it is addressed and may contain information that is
> > > confidential,
> > > > > > privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > > > of this message is not the intended recipient, you are hereby
> > > notified
> > > > > that
> > > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > > forwarding of this communication is strictly prohibited. If you
> > have
> > > > > > received this communication in error, please contact the sender
> > > > > immediately
> > > > > > and delete it from your system. Thank You.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > http://hortonworks.com/download/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Subramaniam V K <su...@gmail.com>.

+1 for end of next week.

We have got all the patches for YARN-1051 committed to the branch. We are
currently fixing test-patch & plan to call a merge vote soon. Hopefully it
will go through by end of next week.

Thanks,
Subru

On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> Looking at the patches, we might be able to get most of YARN-1492 in by the
> end of next week. There would be a couple of security items still
> remaining, but we can may be call the feature alpha-ready without them.
>
> On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:
>
> > I would like to see the shared cache (YARN-1492) make it in as well. We
> are
> > going through the final review process (with Karthik and Vinod) and
> should
> > be fairly close to complete. This is another feature that has been under
> > development for a while.
> >
> > Thanks,
> > Chris
> >
> > On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <suresh@hortonworks.com
> >
> > wrote:
> >
> > > I actually would like to see both archival storage and single replica
> > > memory writes to be in 2.6 release. Archival storage is in the final
> > stages
> > > of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > features are being in development for sometime.
> > >
> > > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <andrew.wang@cloudera.com
> >
> > > wrote:
> > >
> > > > Hey Arun,
> > > >
> > > > Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > > things accordingly?
> > > >
> > > > I think the KMS and transparent encryption are ready to go. We've
> got a
> > > > very few further bug fixes pending, but that's it.
> > > >
> > > > Two HDFS things that I think probably won't make the end of the week
> > are
> > > > archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > > which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > > trunk and I think needs a little more work before it goes into
> > branch-2.
> > > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > > yet.
> > > >
> > > > Just my 2c as I did not work directly on these features. I just
> > generally
> > > > shy away from shipping bits quite this fresh.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >
> > > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > > >
> > > > > I'll create branch-2.6 by end of the week and we can go from there
> to
> > > > > stabilize it - hopefully in the next few weeks.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > thanks,
> > > > > Arun
> > > > >
> > > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com>
> > > > > wrote:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > > > > hadoop-2.6.
> > > > > >
> > > > > >  Currently, here is the Roadmap per the wiki:
> > > > > >
> > > > > >         • HADOOP
> > > > > >                 • Credential provider HADOOP-10607
> > > > > >         • HDFS
> > > > > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > > > using
> > > > > > storage tiers by the applications HDFS-5682
> > > > > >                 • Memory as storage tier HDFS-5851
> > > > > >         • YARN
> > > > > >                 • Dynamic Resource Configuration YARN-291
> > > > > >                 • NodeManager Restart YARN-1336
> > > > > >                 • ResourceManager HA Phase 2 YARN-556
> > > > > >                 • Support for admin-specified labels in YARN
> > YARN-796
> > > > > >                 • Support for automatic, shared cache for YARN
> > > > > application
> > > > > > artifacts YARN-1492
> > > > > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > > > >                 • Support for Docker containers in YARN YARN-1964
> > > > > >                 • YARN service registry YARN-913
> > > > > >
> > > > > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > > > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > > be
> > > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up
> a
> > > > > cadence.
> > > > > >
> > > > > >  More importantly, as we discussed previously, we'd like
> hadoop-2.6
> > > to
> > > > be
> > > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start a
> > > > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > > see
> > > > > > how they feel about this.
> > > > > >
> > > > > > thanks,
> > > > > > Arun
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > --
> > > > > Arun C. Murthy
> > > > > Hortonworks Inc.
> > > > > http://hortonworks.com/
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > http://hortonworks.com/download/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Subramaniam V K <su...@gmail.com>.

+1 for end of next week.

We have got all the patches for YARN-1051 committed to the branch. We are
currently fixing test-patch & plan to call a merge vote soon. Hopefully it
will go through by end of next week.

Thanks,
Subru

On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> Looking at the patches, we might be able to get most of YARN-1492 in by the
> end of next week. There would be a couple of security items still
> remaining, but we can may be call the feature alpha-ready without them.
>
> On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:
>
> > I would like to see the shared cache (YARN-1492) make it in as well. We
> are
> > going through the final review process (with Karthik and Vinod) and
> should
> > be fairly close to complete. This is another feature that has been under
> > development for a while.
> >
> > Thanks,
> > Chris
> >
> > On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <suresh@hortonworks.com
> >
> > wrote:
> >
> > > I actually would like to see both archival storage and single replica
> > > memory writes to be in 2.6 release. Archival storage is in the final
> > stages
> > > of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > features are being in development for sometime.
> > >
> > > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <andrew.wang@cloudera.com
> >
> > > wrote:
> > >
> > > > Hey Arun,
> > > >
> > > > Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > > things accordingly?
> > > >
> > > > I think the KMS and transparent encryption are ready to go. We've
> got a
> > > > very few further bug fixes pending, but that's it.
> > > >
> > > > Two HDFS things that I think probably won't make the end of the week
> > are
> > > > archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > > which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > > trunk and I think needs a little more work before it goes into
> > branch-2.
> > > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > > yet.
> > > >
> > > > Just my 2c as I did not work directly on these features. I just
> > generally
> > > > shy away from shipping bits quite this fresh.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >
> > > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > > >
> > > > > I'll create branch-2.6 by end of the week and we can go from there
> to
> > > > > stabilize it - hopefully in the next few weeks.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > thanks,
> > > > > Arun
> > > > >
> > > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com>
> > > > > wrote:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > > > > hadoop-2.6.
> > > > > >
> > > > > >  Currently, here is the Roadmap per the wiki:
> > > > > >
> > > > > >         • HADOOP
> > > > > >                 • Credential provider HADOOP-10607
> > > > > >         • HDFS
> > > > > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > > > using
> > > > > > storage tiers by the applications HDFS-5682
> > > > > >                 • Memory as storage tier HDFS-5851
> > > > > >         • YARN
> > > > > >                 • Dynamic Resource Configuration YARN-291
> > > > > >                 • NodeManager Restart YARN-1336
> > > > > >                 • ResourceManager HA Phase 2 YARN-556
> > > > > >                 • Support for admin-specified labels in YARN
> > YARN-796
> > > > > >                 • Support for automatic, shared cache for YARN
> > > > > application
> > > > > > artifacts YARN-1492
> > > > > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > > > >                 • Support for Docker containers in YARN YARN-1964
> > > > > >                 • YARN service registry YARN-913
> > > > > >
> > > > > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > > > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > > be
> > > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up
> a
> > > > > cadence.
> > > > > >
> > > > > >  More importantly, as we discussed previously, we'd like
> hadoop-2.6
> > > to
> > > > be
> > > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start a
> > > > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > > see
> > > > > > how they feel about this.
> > > > > >
> > > > > > thanks,
> > > > > > Arun
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > --
> > > > > Arun C. Murthy
> > > > > Hortonworks Inc.
> > > > > http://hortonworks.com/
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > http://hortonworks.com/download/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Subramaniam V K <su...@gmail.com>.

+1 for end of next week.

We have got all the patches for YARN-1051 committed to the branch. We are
currently fixing test-patch & plan to call a merge vote soon. Hopefully it
will go through by end of next week.

Thanks,
Subru

On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> Looking at the patches, we might be able to get most of YARN-1492 in by the
> end of next week. There would be a couple of security items still
> remaining, but we can may be call the feature alpha-ready without them.
>
> On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:
>
> > I would like to see the shared cache (YARN-1492) make it in as well. We
> are
> > going through the final review process (with Karthik and Vinod) and
> should
> > be fairly close to complete. This is another feature that has been under
> > development for a while.
> >
> > Thanks,
> > Chris
> >
> > On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <suresh@hortonworks.com
> >
> > wrote:
> >
> > > I actually would like to see both archival storage and single replica
> > > memory writes to be in 2.6 release. Archival storage is in the final
> > stages
> > > of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > features are being in development for sometime.
> > >
> > > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <andrew.wang@cloudera.com
> >
> > > wrote:
> > >
> > > > Hey Arun,
> > > >
> > > > Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > > things accordingly?
> > > >
> > > > I think the KMS and transparent encryption are ready to go. We've
> got a
> > > > very few further bug fixes pending, but that's it.
> > > >
> > > > Two HDFS things that I think probably won't make the end of the week
> > are
> > > > archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > > which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > > trunk and I think needs a little more work before it goes into
> > branch-2.
> > > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > > yet.
> > > >
> > > > Just my 2c as I did not work directly on these features. I just
> > generally
> > > > shy away from shipping bits quite this fresh.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >
> > > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > > >
> > > > > I'll create branch-2.6 by end of the week and we can go from there
> to
> > > > > stabilize it - hopefully in the next few weeks.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > thanks,
> > > > > Arun
> > > > >
> > > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com>
> > > > > wrote:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > > > > hadoop-2.6.
> > > > > >
> > > > > >  Currently, here is the Roadmap per the wiki:
> > > > > >
> > > > > >         • HADOOP
> > > > > >                 • Credential provider HADOOP-10607
> > > > > >         • HDFS
> > > > > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > > > using
> > > > > > storage tiers by the applications HDFS-5682
> > > > > >                 • Memory as storage tier HDFS-5851
> > > > > >         • YARN
> > > > > >                 • Dynamic Resource Configuration YARN-291
> > > > > >                 • NodeManager Restart YARN-1336
> > > > > >                 • ResourceManager HA Phase 2 YARN-556
> > > > > >                 • Support for admin-specified labels in YARN
> > YARN-796
> > > > > >                 • Support for automatic, shared cache for YARN
> > > > > application
> > > > > > artifacts YARN-1492
> > > > > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > > > >                 • Support for Docker containers in YARN YARN-1964
> > > > > >                 • YARN service registry YARN-913
> > > > > >
> > > > > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > > > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > > be
> > > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up
> a
> > > > > cadence.
> > > > > >
> > > > > >  More importantly, as we discussed previously, we'd like
> hadoop-2.6
> > > to
> > > > be
> > > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start a
> > > > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > > see
> > > > > > how they feel about this.
> > > > > >
> > > > > > thanks,
> > > > > > Arun
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > --
> > > > > Arun C. Murthy
> > > > > Hortonworks Inc.
> > > > > http://hortonworks.com/
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > http://hortonworks.com/download/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Subramaniam V K <su...@gmail.com>.

+1 for end of next week.

We have got all the patches for YARN-1051 committed to the branch. We are
currently fixing test-patch & plan to call a merge vote soon. Hopefully it
will go through by end of next week.

Thanks,
Subru

On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> Looking at the patches, we might be able to get most of YARN-1492 in by the
> end of next week. There would be a couple of security items still
> remaining, but we can may be call the feature alpha-ready without them.
>
> On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:
>
> > I would like to see the shared cache (YARN-1492) make it in as well. We
> are
> > going through the final review process (with Karthik and Vinod) and
> should
> > be fairly close to complete. This is another feature that has been under
> > development for a while.
> >
> > Thanks,
> > Chris
> >
> > On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <suresh@hortonworks.com
> >
> > wrote:
> >
> > > I actually would like to see both archival storage and single replica
> > > memory writes to be in 2.6 release. Archival storage is in the final
> > stages
> > > of getting ready for branch-2 merge as Nicholas has already indicated
> on
> > > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> > these
> > > features are being in development for sometime.
> > >
> > > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <andrew.wang@cloudera.com
> >
> > > wrote:
> > >
> > > > Hey Arun,
> > > >
> > > > Maybe we could do a quick run through of the Roadmap wiki and
> > > add/retarget
> > > > things accordingly?
> > > >
> > > > I think the KMS and transparent encryption are ready to go. We've
> got a
> > > > very few further bug fixes pending, but that's it.
> > > >
> > > > Two HDFS things that I think probably won't make the end of the week
> > are
> > > > archival storage (HDFS-6584) and single replica memory writes
> > > (HDFS-6581),
> > > > which I believe are under the HSM banner. HDFS-6484 was just merged
> to
> > > > trunk and I think needs a little more work before it goes into
> > branch-2.
> > > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
> further
> > > off
> > > > yet.
> > > >
> > > > Just my 2c as I did not work directly on these features. I just
> > generally
> > > > shy away from shipping bits quite this fresh.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >
> > > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > > nicely.
> > > > >
> > > > > I'll create branch-2.6 by end of the week and we can go from there
> to
> > > > > stabilize it - hopefully in the next few weeks.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > thanks,
> > > > > Arun
> > > > >
> > > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
> acm@hortonworks.com>
> > > > > wrote:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead
> to
> > > > > > hadoop-2.6.
> > > > > >
> > > > > >  Currently, here is the Roadmap per the wiki:
> > > > > >
> > > > > >         • HADOOP
> > > > > >                 • Credential provider HADOOP-10607
> > > > > >         • HDFS
> > > > > >                 • Heterogeneous storage (Phase 2) - Support APIs
> > for
> > > > > using
> > > > > > storage tiers by the applications HDFS-5682
> > > > > >                 • Memory as storage tier HDFS-5851
> > > > > >         • YARN
> > > > > >                 • Dynamic Resource Configuration YARN-291
> > > > > >                 • NodeManager Restart YARN-1336
> > > > > >                 • ResourceManager HA Phase 2 YARN-556
> > > > > >                 • Support for admin-specified labels in YARN
> > YARN-796
> > > > > >                 • Support for automatic, shared cache for YARN
> > > > > application
> > > > > > artifacts YARN-1492
> > > > > >                 • Support NodeGroup layer topology on YARN
> YARN-18
> > > > > >                 • Support for Docker containers in YARN YARN-1964
> > > > > >                 • YARN service registry YARN-913
> > > > > >
> > > > > >  My suspicion is, as is normal, some will make the cut and some
> > > won't.
> > > > > > Please do add/subtract from the list as appropriate. Ideally, it
> > > would
> > > > be
> > > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up
> a
> > > > > cadence.
> > > > > >
> > > > > >  More importantly, as we discussed previously, we'd like
> hadoop-2.6
> > > to
> > > > be
> > > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> > start a
> > > > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> > and
> > > > see
> > > > > > how they feel about this.
> > > > > >
> > > > > > thanks,
> > > > > > Arun
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > --
> > > > > Arun C. Murthy
> > > > > Hortonworks Inc.
> > > > > http://hortonworks.com/
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > http://hortonworks.com/download/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Karthik Kambatla <ka...@cloudera.com>.

Looking at the patches, we might be able to get most of YARN-1492 in by the
end of next week. There would be a couple of security items still
remaining, but we can may be call the feature alpha-ready without them.

On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:

> I would like to see the shared cache (YARN-1492) make it in as well. We are
> going through the final review process (with Karthik and Vinod) and should
> be fairly close to complete. This is another feature that has been under
> development for a while.
>
> Thanks,
> Chris
>
> On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
> > I actually would like to see both archival storage and single replica
> > memory writes to be in 2.6 release. Archival storage is in the final
> stages
> > of getting ready for branch-2 merge as Nicholas has already indicated on
> > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> these
> > features are being in development for sometime.
> >
> > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > Hey Arun,
> > >
> > > Maybe we could do a quick run through of the Roadmap wiki and
> > add/retarget
> > > things accordingly?
> > >
> > > I think the KMS and transparent encryption are ready to go. We've got a
> > > very few further bug fixes pending, but that's it.
> > >
> > > Two HDFS things that I think probably won't make the end of the week
> are
> > > archival storage (HDFS-6584) and single replica memory writes
> > (HDFS-6581),
> > > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > > trunk and I think needs a little more work before it goes into
> branch-2.
> > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> > off
> > > yet.
> > >
> > > Just my 2c as I did not work directly on these features. I just
> generally
> > > shy away from shipping bits quite this fresh.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > wrote:
> > >
> > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > nicely.
> > > >
> > > > I'll create branch-2.6 by end of the week and we can go from there to
> > > > stabilize it - hopefully in the next few weeks.
> > > >
> > > > Thoughts?
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > > hadoop-2.6.
> > > > >
> > > > >  Currently, here is the Roadmap per the wiki:
> > > > >
> > > > >         • HADOOP
> > > > >                 • Credential provider HADOOP-10607
> > > > >         • HDFS
> > > > >                 • Heterogeneous storage (Phase 2) - Support APIs
> for
> > > > using
> > > > > storage tiers by the applications HDFS-5682
> > > > >                 • Memory as storage tier HDFS-5851
> > > > >         • YARN
> > > > >                 • Dynamic Resource Configuration YARN-291
> > > > >                 • NodeManager Restart YARN-1336
> > > > >                 • ResourceManager HA Phase 2 YARN-556
> > > > >                 • Support for admin-specified labels in YARN
> YARN-796
> > > > >                 • Support for automatic, shared cache for YARN
> > > > application
> > > > > artifacts YARN-1492
> > > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > > >                 • Support for Docker containers in YARN YARN-1964
> > > > >                 • YARN service registry YARN-913
> > > > >
> > > > >  My suspicion is, as is normal, some will make the cut and some
> > won't.
> > > > > Please do add/subtract from the list as appropriate. Ideally, it
> > would
> > > be
> > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > > cadence.
> > > > >
> > > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> > to
> > > be
> > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> start a
> > > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> and
> > > see
> > > > > how they feel about this.
> > > > >
> > > > > thanks,
> > > > > Arun
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > > Arun C. Murthy
> > > > Hortonworks Inc.
> > > > http://hortonworks.com/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Karthik Kambatla <ka...@cloudera.com>.

Looking at the patches, we might be able to get most of YARN-1492 in by the
end of next week. There would be a couple of security items still
remaining, but we can may be call the feature alpha-ready without them.

On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:

> I would like to see the shared cache (YARN-1492) make it in as well. We are
> going through the final review process (with Karthik and Vinod) and should
> be fairly close to complete. This is another feature that has been under
> development for a while.
>
> Thanks,
> Chris
>
> On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
> > I actually would like to see both archival storage and single replica
> > memory writes to be in 2.6 release. Archival storage is in the final
> stages
> > of getting ready for branch-2 merge as Nicholas has already indicated on
> > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> these
> > features are being in development for sometime.
> >
> > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > Hey Arun,
> > >
> > > Maybe we could do a quick run through of the Roadmap wiki and
> > add/retarget
> > > things accordingly?
> > >
> > > I think the KMS and transparent encryption are ready to go. We've got a
> > > very few further bug fixes pending, but that's it.
> > >
> > > Two HDFS things that I think probably won't make the end of the week
> are
> > > archival storage (HDFS-6584) and single replica memory writes
> > (HDFS-6581),
> > > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > > trunk and I think needs a little more work before it goes into
> branch-2.
> > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> > off
> > > yet.
> > >
> > > Just my 2c as I did not work directly on these features. I just
> generally
> > > shy away from shipping bits quite this fresh.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > wrote:
> > >
> > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > nicely.
> > > >
> > > > I'll create branch-2.6 by end of the week and we can go from there to
> > > > stabilize it - hopefully in the next few weeks.
> > > >
> > > > Thoughts?
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > > hadoop-2.6.
> > > > >
> > > > >  Currently, here is the Roadmap per the wiki:
> > > > >
> > > > >         • HADOOP
> > > > >                 • Credential provider HADOOP-10607
> > > > >         • HDFS
> > > > >                 • Heterogeneous storage (Phase 2) - Support APIs
> for
> > > > using
> > > > > storage tiers by the applications HDFS-5682
> > > > >                 • Memory as storage tier HDFS-5851
> > > > >         • YARN
> > > > >                 • Dynamic Resource Configuration YARN-291
> > > > >                 • NodeManager Restart YARN-1336
> > > > >                 • ResourceManager HA Phase 2 YARN-556
> > > > >                 • Support for admin-specified labels in YARN
> YARN-796
> > > > >                 • Support for automatic, shared cache for YARN
> > > > application
> > > > > artifacts YARN-1492
> > > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > > >                 • Support for Docker containers in YARN YARN-1964
> > > > >                 • YARN service registry YARN-913
> > > > >
> > > > >  My suspicion is, as is normal, some will make the cut and some
> > won't.
> > > > > Please do add/subtract from the list as appropriate. Ideally, it
> > would
> > > be
> > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > > cadence.
> > > > >
> > > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> > to
> > > be
> > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> start a
> > > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> and
> > > see
> > > > > how they feel about this.
> > > > >
> > > > > thanks,
> > > > > Arun
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > > Arun C. Murthy
> > > > Hortonworks Inc.
> > > > http://hortonworks.com/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Karthik Kambatla <ka...@cloudera.com>.

Looking at the patches, we might be able to get most of YARN-1492 in by the
end of next week. There would be a couple of security items still
remaining, but we can may be call the feature alpha-ready without them.

On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:

> I would like to see the shared cache (YARN-1492) make it in as well. We are
> going through the final review process (with Karthik and Vinod) and should
> be fairly close to complete. This is another feature that has been under
> development for a while.
>
> Thanks,
> Chris
>
> On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
> > I actually would like to see both archival storage and single replica
> > memory writes to be in 2.6 release. Archival storage is in the final
> stages
> > of getting ready for branch-2 merge as Nicholas has already indicated on
> > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> these
> > features are being in development for sometime.
> >
> > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > Hey Arun,
> > >
> > > Maybe we could do a quick run through of the Roadmap wiki and
> > add/retarget
> > > things accordingly?
> > >
> > > I think the KMS and transparent encryption are ready to go. We've got a
> > > very few further bug fixes pending, but that's it.
> > >
> > > Two HDFS things that I think probably won't make the end of the week
> are
> > > archival storage (HDFS-6584) and single replica memory writes
> > (HDFS-6581),
> > > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > > trunk and I think needs a little more work before it goes into
> branch-2.
> > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> > off
> > > yet.
> > >
> > > Just my 2c as I did not work directly on these features. I just
> generally
> > > shy away from shipping bits quite this fresh.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > wrote:
> > >
> > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > nicely.
> > > >
> > > > I'll create branch-2.6 by end of the week and we can go from there to
> > > > stabilize it - hopefully in the next few weeks.
> > > >
> > > > Thoughts?
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > > hadoop-2.6.
> > > > >
> > > > >  Currently, here is the Roadmap per the wiki:
> > > > >
> > > > >         • HADOOP
> > > > >                 • Credential provider HADOOP-10607
> > > > >         • HDFS
> > > > >                 • Heterogeneous storage (Phase 2) - Support APIs
> for
> > > > using
> > > > > storage tiers by the applications HDFS-5682
> > > > >                 • Memory as storage tier HDFS-5851
> > > > >         • YARN
> > > > >                 • Dynamic Resource Configuration YARN-291
> > > > >                 • NodeManager Restart YARN-1336
> > > > >                 • ResourceManager HA Phase 2 YARN-556
> > > > >                 • Support for admin-specified labels in YARN
> YARN-796
> > > > >                 • Support for automatic, shared cache for YARN
> > > > application
> > > > > artifacts YARN-1492
> > > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > > >                 • Support for Docker containers in YARN YARN-1964
> > > > >                 • YARN service registry YARN-913
> > > > >
> > > > >  My suspicion is, as is normal, some will make the cut and some
> > won't.
> > > > > Please do add/subtract from the list as appropriate. Ideally, it
> > would
> > > be
> > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > > cadence.
> > > > >
> > > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> > to
> > > be
> > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> start a
> > > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> and
> > > see
> > > > > how they feel about this.
> > > > >
> > > > > thanks,
> > > > > Arun
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > > Arun C. Murthy
> > > > Hortonworks Inc.
> > > > http://hortonworks.com/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Karthik Kambatla <ka...@cloudera.com>.

Looking at the patches, we might be able to get most of YARN-1492 in by the
end of next week. There would be a couple of security items still
remaining, but we can may be call the feature alpha-ready without them.

On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo <ct...@gmail.com> wrote:

> I would like to see the shared cache (YARN-1492) make it in as well. We are
> going through the final review process (with Karthik and Vinod) and should
> be fairly close to complete. This is another feature that has been under
> development for a while.
>
> Thanks,
> Chris
>
> On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
> > I actually would like to see both archival storage and single replica
> > memory writes to be in 2.6 release. Archival storage is in the final
> stages
> > of getting ready for branch-2 merge as Nicholas has already indicated on
> > the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
> these
> > features are being in development for sometime.
> >
> > On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > Hey Arun,
> > >
> > > Maybe we could do a quick run through of the Roadmap wiki and
> > add/retarget
> > > things accordingly?
> > >
> > > I think the KMS and transparent encryption are ready to go. We've got a
> > > very few further bug fixes pending, but that's it.
> > >
> > > Two HDFS things that I think probably won't make the end of the week
> are
> > > archival storage (HDFS-6584) and single replica memory writes
> > (HDFS-6581),
> > > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > > trunk and I think needs a little more work before it goes into
> branch-2.
> > > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> > off
> > > yet.
> > >
> > > Just my 2c as I did not work directly on these features. I just
> generally
> > > shy away from shipping bits quite this fresh.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> > wrote:
> > >
> > > > Looks like most of the content is in and hadoop-2.6 is shaping up
> > nicely.
> > > >
> > > > I'll create branch-2.6 by end of the week and we can go from there to
> > > > stabilize it - hopefully in the next few weeks.
> > > >
> > > > Thoughts?
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > > hadoop-2.6.
> > > > >
> > > > >  Currently, here is the Roadmap per the wiki:
> > > > >
> > > > >         • HADOOP
> > > > >                 • Credential provider HADOOP-10607
> > > > >         • HDFS
> > > > >                 • Heterogeneous storage (Phase 2) - Support APIs
> for
> > > > using
> > > > > storage tiers by the applications HDFS-5682
> > > > >                 • Memory as storage tier HDFS-5851
> > > > >         • YARN
> > > > >                 • Dynamic Resource Configuration YARN-291
> > > > >                 • NodeManager Restart YARN-1336
> > > > >                 • ResourceManager HA Phase 2 YARN-556
> > > > >                 • Support for admin-specified labels in YARN
> YARN-796
> > > > >                 • Support for automatic, shared cache for YARN
> > > > application
> > > > > artifacts YARN-1492
> > > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > > >                 • Support for Docker containers in YARN YARN-1964
> > > > >                 • YARN service registry YARN-913
> > > > >
> > > > >  My suspicion is, as is normal, some will make the cut and some
> > won't.
> > > > > Please do add/subtract from the list as appropriate. Ideally, it
> > would
> > > be
> > > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > > cadence.
> > > > >
> > > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> > to
> > > be
> > > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll
> start a
> > > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.)
> and
> > > see
> > > > > how they feel about this.
> > > > >
> > > > > thanks,
> > > > > Arun
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > > Arun C. Murthy
> > > > Hortonworks Inc.
> > > > http://hortonworks.com/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

Re: Thinking ahead to hadoop-2.6

Posted by Chris Trezzo <ct...@gmail.com>.

I would like to see the shared cache (YARN-1492) make it in as well. We are
going through the final review process (with Karthik and Vinod) and should
be fairly close to complete. This is another feature that has been under
development for a while.

Thanks,
Chris

On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> I actually would like to see both archival storage and single replica
> memory writes to be in 2.6 release. Archival storage is in the final stages
> of getting ready for branch-2 merge as Nicholas has already indicated on
> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> features are being in development for sometime.
>
> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Arun,
> >
> > Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> > things accordingly?
> >
> > I think the KMS and transparent encryption are ready to go. We've got a
> > very few further bug fixes pending, but that's it.
> >
> > Two HDFS things that I think probably won't make the end of the week are
> > archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > trunk and I think needs a little more work before it goes into branch-2.
> > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> > yet.
> >
> > Just my 2c as I did not work directly on these features. I just generally
> > shy away from shipping bits quite this fresh.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >
> > > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> > >
> > > I'll create branch-2.6 by end of the week and we can go from there to
> > > stabilize it - hopefully in the next few weeks.
> > >
> > > Thoughts?
> > >
> > > thanks,
> > > Arun
> > >
> > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > hadoop-2.6.
> > > >
> > > >  Currently, here is the Roadmap per the wiki:
> > > >
> > > >         • HADOOP
> > > >                 • Credential provider HADOOP-10607
> > > >         • HDFS
> > > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > > using
> > > > storage tiers by the applications HDFS-5682
> > > >                 • Memory as storage tier HDFS-5851
> > > >         • YARN
> > > >                 • Dynamic Resource Configuration YARN-291
> > > >                 • NodeManager Restart YARN-1336
> > > >                 • ResourceManager HA Phase 2 YARN-556
> > > >                 • Support for admin-specified labels in YARN YARN-796
> > > >                 • Support for automatic, shared cache for YARN
> > > application
> > > > artifacts YARN-1492
> > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > >                 • Support for Docker containers in YARN YARN-1964
> > > >                 • YARN service registry YARN-913
> > > >
> > > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> > > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> > be
> > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > cadence.
> > > >
> > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> > be
> > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> > see
> > > > how they feel about this.
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by "Tsz Wo (Nicholas), Sze" <s2...@yahoo.com.INVALID>.

Hi,

I am worry about KMS and transparent encryption since there are quite many bugs discovered after it got merged to branch-2.  It gives us an impression that the feature is not yet well tested.  Indeed, transparent encryption is a complicated feature which changes the core part of HDFS.  It is not easy to get everything right.


For HDFS-6584: Archival Storage, it is a relatively simple and low risk feature.  It introduces a new storage type ARCHIVE and the concept of block storage policy to HDFS.  When a cluster is configured with ARCHIVE storage, the blocks will be stored using the appropriate storage types specified by storage policies assigned to the files/directories.  Cluster admin could disable the feature by simply not configuring any storage type and not setting any storage policy as before.   As Suresh mentioned, HDFS-6584 is in the final stages to be merged to branch-2.

Regards,
Tsz-Wo



On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <su...@hortonworks.com> wrote:
 

>
>
>I actually would like to see both archival storage and single replica
>memory writes to be in 2.6 release. Archival storage is in the final stages
>of getting ready for branch-2 merge as Nicholas has already indicated on
>the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
>features are being in development for sometime.
>
>On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
>wrote:
>
>> Hey Arun,
>>
>> Maybe we could do a quick run through of the Roadmap wiki and add/retarget
>> things accordingly?
>>
>> I think the KMS and transparent encryption are ready to go. We've got a
>> very few further bug fixes pending, but that's it.
>>
>> Two HDFS things that I think probably won't make the end of the week are
>> archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
>> which I believe are under the HSM banner. HDFS-6484 was just merged to
>> trunk and I think needs a little more work before it goes into branch-2.
>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
>> yet.
>>
>> Just my 2c as I did not work directly on these features. I just generally
>> shy away from shipping bits quite this fresh.
>>
>> Thanks,
>> Andrew
>>
>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:
>>
>> > Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>> >
>> > I'll create branch-2.6 by end of the week and we can go from there to
>> > stabilize it - hopefully in the next few weeks.
>> >
>> > Thoughts?
>> >
>> > thanks,
>> > Arun
>> >
>> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
>> > wrote:
>> >
>> > > Folks,
>> > >
>> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
>> > > hadoop-2.6.
>> > >
>> > >  Currently, here is the Roadmap per the wiki:
>> > >
>> > >         • HADOOP
>> > >                 • Credential provider HADOOP-10607
>> > >         • HDFS
>> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
>> > using
>> > > storage tiers by the applications HDFS-5682
>> > >                 • Memory as storage tier HDFS-5851
>> > >         • YARN
>> > >                 • Dynamic Resource Configuration YARN-291
>> > >                 • NodeManager Restart YARN-1336
>> > >                 • ResourceManager HA Phase 2 YARN-556
>> > >                 • Support for admin-specified labels in YARN YARN-796
>> > >                 • Support for automatic, shared cache for YARN
>> > application
>> > > artifacts YARN-1492
>> > >                 • Support NodeGroup layer topology on YARN YARN-18
>> > >                 • Support for Docker containers in YARN YARN-1964
>> > >                 • YARN service registry YARN-913
>> > >
>> > >  My suspicion is, as is normal, some will make the cut and some won't.
>> > > Please do add/subtract from the list as appropriate. Ideally, it would
>> be
>> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
>> > cadence.
>> > >
>> > >  More importantly, as we discussed previously, we'd like hadoop-2.6 to
>> be
>> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
>> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
>> see
>> > > how they feel about this.
>> > >
>> > > thanks,
>> > > Arun
>> > >
>> > >
>> >
>> >
>> > --
>> >
>> > --
>> > Arun C. Murthy
>> > Hortonworks Inc.
>> > http://hortonworks.com/
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>
>
>-- 
>http://hortonworks.com/download/
>
>-- 
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity to 
>which it is addressed and may contain information that is confidential, 
>privileged and exempt from disclosure under applicable law. If the reader 
>of this message is not the intended recipient, you are hereby notified that 
>any printing, copying, dissemination, distribution, disclosure or 
>forwarding of this communication is strictly prohibited. If you have 
>received this communication in error, please contact the sender immediately 
>and delete it from your system. Thank You.
>
>

Re: Thinking ahead to hadoop-2.6

Posted by Chris Trezzo <ct...@gmail.com>.

I would like to see the shared cache (YARN-1492) make it in as well. We are
going through the final review process (with Karthik and Vinod) and should
be fairly close to complete. This is another feature that has been under
development for a while.

Thanks,
Chris

On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> I actually would like to see both archival storage and single replica
> memory writes to be in 2.6 release. Archival storage is in the final stages
> of getting ready for branch-2 merge as Nicholas has already indicated on
> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> features are being in development for sometime.
>
> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Arun,
> >
> > Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> > things accordingly?
> >
> > I think the KMS and transparent encryption are ready to go. We've got a
> > very few further bug fixes pending, but that's it.
> >
> > Two HDFS things that I think probably won't make the end of the week are
> > archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > trunk and I think needs a little more work before it goes into branch-2.
> > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> > yet.
> >
> > Just my 2c as I did not work directly on these features. I just generally
> > shy away from shipping bits quite this fresh.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >
> > > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> > >
> > > I'll create branch-2.6 by end of the week and we can go from there to
> > > stabilize it - hopefully in the next few weeks.
> > >
> > > Thoughts?
> > >
> > > thanks,
> > > Arun
> > >
> > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > hadoop-2.6.
> > > >
> > > >  Currently, here is the Roadmap per the wiki:
> > > >
> > > >         • HADOOP
> > > >                 • Credential provider HADOOP-10607
> > > >         • HDFS
> > > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > > using
> > > > storage tiers by the applications HDFS-5682
> > > >                 • Memory as storage tier HDFS-5851
> > > >         • YARN
> > > >                 • Dynamic Resource Configuration YARN-291
> > > >                 • NodeManager Restart YARN-1336
> > > >                 • ResourceManager HA Phase 2 YARN-556
> > > >                 • Support for admin-specified labels in YARN YARN-796
> > > >                 • Support for automatic, shared cache for YARN
> > > application
> > > > artifacts YARN-1492
> > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > >                 • Support for Docker containers in YARN YARN-1964
> > > >                 • YARN service registry YARN-913
> > > >
> > > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> > > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> > be
> > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > cadence.
> > > >
> > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> > be
> > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> > see
> > > > how they feel about this.
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Chris Trezzo <ct...@gmail.com>.

I would like to see the shared cache (YARN-1492) make it in as well. We are
going through the final review process (with Karthik and Vinod) and should
be fairly close to complete. This is another feature that has been under
development for a while.

Thanks,
Chris

On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> I actually would like to see both archival storage and single replica
> memory writes to be in 2.6 release. Archival storage is in the final stages
> of getting ready for branch-2 merge as Nicholas has already indicated on
> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> features are being in development for sometime.
>
> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Arun,
> >
> > Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> > things accordingly?
> >
> > I think the KMS and transparent encryption are ready to go. We've got a
> > very few further bug fixes pending, but that's it.
> >
> > Two HDFS things that I think probably won't make the end of the week are
> > archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > trunk and I think needs a little more work before it goes into branch-2.
> > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> > yet.
> >
> > Just my 2c as I did not work directly on these features. I just generally
> > shy away from shipping bits quite this fresh.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >
> > > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> > >
> > > I'll create branch-2.6 by end of the week and we can go from there to
> > > stabilize it - hopefully in the next few weeks.
> > >
> > > Thoughts?
> > >
> > > thanks,
> > > Arun
> > >
> > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > hadoop-2.6.
> > > >
> > > >  Currently, here is the Roadmap per the wiki:
> > > >
> > > >         • HADOOP
> > > >                 • Credential provider HADOOP-10607
> > > >         • HDFS
> > > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > > using
> > > > storage tiers by the applications HDFS-5682
> > > >                 • Memory as storage tier HDFS-5851
> > > >         • YARN
> > > >                 • Dynamic Resource Configuration YARN-291
> > > >                 • NodeManager Restart YARN-1336
> > > >                 • ResourceManager HA Phase 2 YARN-556
> > > >                 • Support for admin-specified labels in YARN YARN-796
> > > >                 • Support for automatic, shared cache for YARN
> > > application
> > > > artifacts YARN-1492
> > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > >                 • Support for Docker containers in YARN YARN-1964
> > > >                 • YARN service registry YARN-913
> > > >
> > > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> > > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> > be
> > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > cadence.
> > > >
> > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> > be
> > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> > see
> > > > how they feel about this.
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Chris Trezzo <ct...@gmail.com>.

I would like to see the shared cache (YARN-1492) make it in as well. We are
going through the final review process (with Karthik and Vinod) and should
be fairly close to complete. This is another feature that has been under
development for a while.

Thanks,
Chris

On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas <su...@hortonworks.com>
wrote:

> I actually would like to see both archival storage and single replica
> memory writes to be in 2.6 release. Archival storage is in the final stages
> of getting ready for branch-2 merge as Nicholas has already indicated on
> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
> features are being in development for sometime.
>
> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hey Arun,
> >
> > Maybe we could do a quick run through of the Roadmap wiki and
> add/retarget
> > things accordingly?
> >
> > I think the KMS and transparent encryption are ready to go. We've got a
> > very few further bug fixes pending, but that's it.
> >
> > Two HDFS things that I think probably won't make the end of the week are
> > archival storage (HDFS-6584) and single replica memory writes
> (HDFS-6581),
> > which I believe are under the HSM banner. HDFS-6484 was just merged to
> > trunk and I think needs a little more work before it goes into branch-2.
> > HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further
> off
> > yet.
> >
> > Just my 2c as I did not work directly on these features. I just generally
> > shy away from shipping bits quite this fresh.
> >
> > Thanks,
> > Andrew
> >
> > On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
> wrote:
> >
> > > Looks like most of the content is in and hadoop-2.6 is shaping up
> nicely.
> > >
> > > I'll create branch-2.6 by end of the week and we can go from there to
> > > stabilize it - hopefully in the next few weeks.
> > >
> > > Thoughts?
> > >
> > > thanks,
> > > Arun
> > >
> > > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > > hadoop-2.6.
> > > >
> > > >  Currently, here is the Roadmap per the wiki:
> > > >
> > > >         • HADOOP
> > > >                 • Credential provider HADOOP-10607
> > > >         • HDFS
> > > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > > using
> > > > storage tiers by the applications HDFS-5682
> > > >                 • Memory as storage tier HDFS-5851
> > > >         • YARN
> > > >                 • Dynamic Resource Configuration YARN-291
> > > >                 • NodeManager Restart YARN-1336
> > > >                 • ResourceManager HA Phase 2 YARN-556
> > > >                 • Support for admin-specified labels in YARN YARN-796
> > > >                 • Support for automatic, shared cache for YARN
> > > application
> > > > artifacts YARN-1492
> > > >                 • Support NodeGroup layer topology on YARN YARN-18
> > > >                 • Support for Docker containers in YARN YARN-1964
> > > >                 • YARN service registry YARN-913
> > > >
> > > >  My suspicion is, as is normal, some will make the cut and some
> won't.
> > > > Please do add/subtract from the list as appropriate. Ideally, it
> would
> > be
> > > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > > cadence.
> > > >
> > > >  More importantly, as we discussed previously, we'd like hadoop-2.6
> to
> > be
> > > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> > see
> > > > how they feel about this.
> > > >
> > > > thanks,
> > > > Arun
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Suresh Srinivas <su...@hortonworks.com>.

I actually would like to see both archival storage and single replica
memory writes to be in 2.6 release. Archival storage is in the final stages
of getting ready for branch-2 merge as Nicholas has already indicated on
the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
features are being in development for sometime.

On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
wrote:

> Hey Arun,
>
> Maybe we could do a quick run through of the Roadmap wiki and add/retarget
> things accordingly?
>
> I think the KMS and transparent encryption are ready to go. We've got a
> very few further bug fixes pending, but that's it.
>
> Two HDFS things that I think probably won't make the end of the week are
> archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
> which I believe are under the HSM banner. HDFS-6484 was just merged to
> trunk and I think needs a little more work before it goes into branch-2.
> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
> yet.
>
> Just my 2c as I did not work directly on these features. I just generally
> shy away from shipping bits quite this fresh.
>
> Thanks,
> Andrew
>
> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:
>
> > Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
> >
> > I'll create branch-2.6 by end of the week and we can go from there to
> > stabilize it - hopefully in the next few weeks.
> >
> > Thoughts?
> >
> > thanks,
> > Arun
> >
> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > wrote:
> >
> > > Folks,
> > >
> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > hadoop-2.6.
> > >
> > >  Currently, here is the Roadmap per the wiki:
> > >
> > >         • HADOOP
> > >                 • Credential provider HADOOP-10607
> > >         • HDFS
> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > using
> > > storage tiers by the applications HDFS-5682
> > >                 • Memory as storage tier HDFS-5851
> > >         • YARN
> > >                 • Dynamic Resource Configuration YARN-291
> > >                 • NodeManager Restart YARN-1336
> > >                 • ResourceManager HA Phase 2 YARN-556
> > >                 • Support for admin-specified labels in YARN YARN-796
> > >                 • Support for automatic, shared cache for YARN
> > application
> > > artifacts YARN-1492
> > >                 • Support NodeGroup layer topology on YARN YARN-18
> > >                 • Support for Docker containers in YARN YARN-1964
> > >                 • YARN service registry YARN-913
> > >
> > >  My suspicion is, as is normal, some will make the cut and some won't.
> > > Please do add/subtract from the list as appropriate. Ideally, it would
> be
> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > cadence.
> > >
> > >  More importantly, as we discussed previously, we'd like hadoop-2.6 to
> be
> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> see
> > > how they feel about this.
> > >
> > > thanks,
> > > Arun
> > >
> > >
> >
> >
> > --
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>



-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Suresh Srinivas <su...@hortonworks.com>.

I actually would like to see both archival storage and single replica
memory writes to be in 2.6 release. Archival storage is in the final stages
of getting ready for branch-2 merge as Nicholas has already indicated on
the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
features are being in development for sometime.

On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
wrote:

> Hey Arun,
>
> Maybe we could do a quick run through of the Roadmap wiki and add/retarget
> things accordingly?
>
> I think the KMS and transparent encryption are ready to go. We've got a
> very few further bug fixes pending, but that's it.
>
> Two HDFS things that I think probably won't make the end of the week are
> archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
> which I believe are under the HSM banner. HDFS-6484 was just merged to
> trunk and I think needs a little more work before it goes into branch-2.
> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
> yet.
>
> Just my 2c as I did not work directly on these features. I just generally
> shy away from shipping bits quite this fresh.
>
> Thanks,
> Andrew
>
> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:
>
> > Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
> >
> > I'll create branch-2.6 by end of the week and we can go from there to
> > stabilize it - hopefully in the next few weeks.
> >
> > Thoughts?
> >
> > thanks,
> > Arun
> >
> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > wrote:
> >
> > > Folks,
> > >
> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > hadoop-2.6.
> > >
> > >  Currently, here is the Roadmap per the wiki:
> > >
> > >         • HADOOP
> > >                 • Credential provider HADOOP-10607
> > >         • HDFS
> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > using
> > > storage tiers by the applications HDFS-5682
> > >                 • Memory as storage tier HDFS-5851
> > >         • YARN
> > >                 • Dynamic Resource Configuration YARN-291
> > >                 • NodeManager Restart YARN-1336
> > >                 • ResourceManager HA Phase 2 YARN-556
> > >                 • Support for admin-specified labels in YARN YARN-796
> > >                 • Support for automatic, shared cache for YARN
> > application
> > > artifacts YARN-1492
> > >                 • Support NodeGroup layer topology on YARN YARN-18
> > >                 • Support for Docker containers in YARN YARN-1964
> > >                 • YARN service registry YARN-913
> > >
> > >  My suspicion is, as is normal, some will make the cut and some won't.
> > > Please do add/subtract from the list as appropriate. Ideally, it would
> be
> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > cadence.
> > >
> > >  More importantly, as we discussed previously, we'd like hadoop-2.6 to
> be
> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> see
> > > how they feel about this.
> > >
> > > thanks,
> > > Arun
> > >
> > >
> >
> >
> > --
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>



-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Suresh Srinivas <su...@hortonworks.com>.

I actually would like to see both archival storage and single replica
memory writes to be in 2.6 release. Archival storage is in the final stages
of getting ready for branch-2 merge as Nicholas has already indicated on
the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
features are being in development for sometime.

On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
wrote:

> Hey Arun,
>
> Maybe we could do a quick run through of the Roadmap wiki and add/retarget
> things accordingly?
>
> I think the KMS and transparent encryption are ready to go. We've got a
> very few further bug fixes pending, but that's it.
>
> Two HDFS things that I think probably won't make the end of the week are
> archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
> which I believe are under the HSM banner. HDFS-6484 was just merged to
> trunk and I think needs a little more work before it goes into branch-2.
> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
> yet.
>
> Just my 2c as I did not work directly on these features. I just generally
> shy away from shipping bits quite this fresh.
>
> Thanks,
> Andrew
>
> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:
>
> > Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
> >
> > I'll create branch-2.6 by end of the week and we can go from there to
> > stabilize it - hopefully in the next few weeks.
> >
> > Thoughts?
> >
> > thanks,
> > Arun
> >
> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > wrote:
> >
> > > Folks,
> > >
> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > hadoop-2.6.
> > >
> > >  Currently, here is the Roadmap per the wiki:
> > >
> > >         • HADOOP
> > >                 • Credential provider HADOOP-10607
> > >         • HDFS
> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > using
> > > storage tiers by the applications HDFS-5682
> > >                 • Memory as storage tier HDFS-5851
> > >         • YARN
> > >                 • Dynamic Resource Configuration YARN-291
> > >                 • NodeManager Restart YARN-1336
> > >                 • ResourceManager HA Phase 2 YARN-556
> > >                 • Support for admin-specified labels in YARN YARN-796
> > >                 • Support for automatic, shared cache for YARN
> > application
> > > artifacts YARN-1492
> > >                 • Support NodeGroup layer topology on YARN YARN-18
> > >                 • Support for Docker containers in YARN YARN-1964
> > >                 • YARN service registry YARN-913
> > >
> > >  My suspicion is, as is normal, some will make the cut and some won't.
> > > Please do add/subtract from the list as appropriate. Ideally, it would
> be
> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > cadence.
> > >
> > >  More importantly, as we discussed previously, we'd like hadoop-2.6 to
> be
> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> see
> > > how they feel about this.
> > >
> > > thanks,
> > > Arun
> > >
> > >
> >
> >
> > --
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>



-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Suresh Srinivas <su...@hortonworks.com>.

I actually would like to see both archival storage and single replica
memory writes to be in 2.6 release. Archival storage is in the final stages
of getting ready for branch-2 merge as Nicholas has already indicated on
the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these
features are being in development for sometime.

On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <an...@cloudera.com>
wrote:

> Hey Arun,
>
> Maybe we could do a quick run through of the Roadmap wiki and add/retarget
> things accordingly?
>
> I think the KMS and transparent encryption are ready to go. We've got a
> very few further bug fixes pending, but that's it.
>
> Two HDFS things that I think probably won't make the end of the week are
> archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
> which I believe are under the HSM banner. HDFS-6484 was just merged to
> trunk and I think needs a little more work before it goes into branch-2.
> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
> yet.
>
> Just my 2c as I did not work directly on these features. I just generally
> shy away from shipping bits quite this fresh.
>
> Thanks,
> Andrew
>
> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:
>
> > Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
> >
> > I'll create branch-2.6 by end of the week and we can go from there to
> > stabilize it - hopefully in the next few weeks.
> >
> > Thoughts?
> >
> > thanks,
> > Arun
> >
> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> > wrote:
> >
> > > Folks,
> > >
> > >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > > hadoop-2.6.
> > >
> > >  Currently, here is the Roadmap per the wiki:
> > >
> > >         • HADOOP
> > >                 • Credential provider HADOOP-10607
> > >         • HDFS
> > >                 • Heterogeneous storage (Phase 2) - Support APIs for
> > using
> > > storage tiers by the applications HDFS-5682
> > >                 • Memory as storage tier HDFS-5851
> > >         • YARN
> > >                 • Dynamic Resource Configuration YARN-291
> > >                 • NodeManager Restart YARN-1336
> > >                 • ResourceManager HA Phase 2 YARN-556
> > >                 • Support for admin-specified labels in YARN YARN-796
> > >                 • Support for automatic, shared cache for YARN
> > application
> > > artifacts YARN-1492
> > >                 • Support NodeGroup layer topology on YARN YARN-18
> > >                 • Support for Docker containers in YARN YARN-1964
> > >                 • YARN service registry YARN-913
> > >
> > >  My suspicion is, as is normal, some will make the cut and some won't.
> > > Please do add/subtract from the list as appropriate. Ideally, it would
> be
> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> > cadence.
> > >
> > >  More importantly, as we discussed previously, we'd like hadoop-2.6 to
> be
> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and
> see
> > > how they feel about this.
> > >
> > > thanks,
> > > Arun
> > >
> > >
> >
> >
> > --
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>



-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

Hey Arun,

Maybe we could do a quick run through of the Roadmap wiki and add/retarget
things accordingly?

I think the KMS and transparent encryption are ready to go. We've got a
very few further bug fixes pending, but that's it.

Two HDFS things that I think probably won't make the end of the week are
archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
which I believe are under the HSM banner. HDFS-6484 was just merged to
trunk and I think needs a little more work before it goes into branch-2.
HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
yet.

Just my 2c as I did not work directly on these features. I just generally
shy away from shipping bits quite this fresh.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:

> Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>
> I'll create branch-2.6 by end of the week and we can go from there to
> stabilize it - hopefully in the next few weeks.
>
> Thoughts?
>
> thanks,
> Arun
>
> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
>
> > Folks,
> >
> >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >  Currently, here is the Roadmap per the wiki:
> >
> >         • HADOOP
> >                 • Credential provider HADOOP-10607
> >         • HDFS
> >                 • Heterogeneous storage (Phase 2) - Support APIs for
> using
> > storage tiers by the applications HDFS-5682
> >                 • Memory as storage tier HDFS-5851
> >         • YARN
> >                 • Dynamic Resource Configuration YARN-291
> >                 • NodeManager Restart YARN-1336
> >                 • ResourceManager HA Phase 2 YARN-556
> >                 • Support for admin-specified labels in YARN YARN-796
> >                 • Support for automatic, shared cache for YARN
> application
> > artifacts YARN-1492
> >                 • Support NodeGroup layer topology on YARN YARN-18
> >                 • Support for Docker containers in YARN YARN-1964
> >                 • YARN service registry YARN-913
> >
> >  My suspicion is, as is normal, some will make the cut and some won't.
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> > thanks,
> > Arun
> >
> >
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

Hey Arun,

Maybe we could do a quick run through of the Roadmap wiki and add/retarget
things accordingly?

I think the KMS and transparent encryption are ready to go. We've got a
very few further bug fixes pending, but that's it.

Two HDFS things that I think probably won't make the end of the week are
archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
which I believe are under the HSM banner. HDFS-6484 was just merged to
trunk and I think needs a little more work before it goes into branch-2.
HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
yet.

Just my 2c as I did not work directly on these features. I just generally
shy away from shipping bits quite this fresh.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:

> Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>
> I'll create branch-2.6 by end of the week and we can go from there to
> stabilize it - hopefully in the next few weeks.
>
> Thoughts?
>
> thanks,
> Arun
>
> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
>
> > Folks,
> >
> >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >  Currently, here is the Roadmap per the wiki:
> >
> >         • HADOOP
> >                 • Credential provider HADOOP-10607
> >         • HDFS
> >                 • Heterogeneous storage (Phase 2) - Support APIs for
> using
> > storage tiers by the applications HDFS-5682
> >                 • Memory as storage tier HDFS-5851
> >         • YARN
> >                 • Dynamic Resource Configuration YARN-291
> >                 • NodeManager Restart YARN-1336
> >                 • ResourceManager HA Phase 2 YARN-556
> >                 • Support for admin-specified labels in YARN YARN-796
> >                 • Support for automatic, shared cache for YARN
> application
> > artifacts YARN-1492
> >                 • Support NodeGroup layer topology on YARN YARN-18
> >                 • Support for Docker containers in YARN YARN-1964
> >                 • YARN service registry YARN-913
> >
> >  My suspicion is, as is normal, some will make the cut and some won't.
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> > thanks,
> > Arun
> >
> >
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

Hey Arun,

Maybe we could do a quick run through of the Roadmap wiki and add/retarget
things accordingly?

I think the KMS and transparent encryption are ready to go. We've got a
very few further bug fixes pending, but that's it.

Two HDFS things that I think probably won't make the end of the week are
archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
which I believe are under the HSM banner. HDFS-6484 was just merged to
trunk and I think needs a little more work before it goes into branch-2.
HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
yet.

Just my 2c as I did not work directly on these features. I just generally
shy away from shipping bits quite this fresh.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:

> Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>
> I'll create branch-2.6 by end of the week and we can go from there to
> stabilize it - hopefully in the next few weeks.
>
> Thoughts?
>
> thanks,
> Arun
>
> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
>
> > Folks,
> >
> >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >  Currently, here is the Roadmap per the wiki:
> >
> >         • HADOOP
> >                 • Credential provider HADOOP-10607
> >         • HDFS
> >                 • Heterogeneous storage (Phase 2) - Support APIs for
> using
> > storage tiers by the applications HDFS-5682
> >                 • Memory as storage tier HDFS-5851
> >         • YARN
> >                 • Dynamic Resource Configuration YARN-291
> >                 • NodeManager Restart YARN-1336
> >                 • ResourceManager HA Phase 2 YARN-556
> >                 • Support for admin-specified labels in YARN YARN-796
> >                 • Support for automatic, shared cache for YARN
> application
> > artifacts YARN-1492
> >                 • Support NodeGroup layer topology on YARN YARN-18
> >                 • Support for Docker containers in YARN YARN-1964
> >                 • YARN service registry YARN-913
> >
> >  My suspicion is, as is normal, some will make the cut and some won't.
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> > thanks,
> > Arun
> >
> >
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.

Thank for restarting this, Arun!

The following are the efforts that I am involved in directly or indirectly
and shepherding from YARN's point of view: rolling-upgrades (YARN-666),
work-preserving RM restart (YARN-556), minimal long-running services
support (YARN-896), including service-registry (YARN-913), timeline-service
stability/security (some sub-tasks of YARN-1935), node labels (YARN-296)
and some enhancements/bug fixes in cpu-scheduling/cgroups. We also have
reservations sub-system (YARN-1051) and  YARN shared cache (YARN-1492) that
I would like to get in and helping reviews with. Almost all these efforts
are on the edge of completion - some of them are just pending reviews.
Clearly some of them will be more stable than others, some of them make it
eventually, some may not.

+1 for branching it within a week so that we can stabilize it and let other
features go into branch-2, I am putting out all the stops to get the above
YARN efforts to be in reasonable shape for 2.6.

Regarding remaining items that I haven't paid much attention, NM restart
effort - mostly done by Jason Lowe - YARN-1336 is all in. YARN-291,
YARN-18, YARN-1964 are not going to cut from what I see.

I edited the road-map to reflect this latest reality.

+Vinod

On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:

> Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>
> I'll create branch-2.6 by end of the week and we can go from there to
> stabilize it - hopefully in the next few weeks.
>
> Thoughts?
>
> thanks,
> Arun
>
> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
>
> > Folks,
> >
> >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >  Currently, here is the Roadmap per the wiki:
> >
> >         • HADOOP
> >                 • Credential provider HADOOP-10607
> >         • HDFS
> >                 • Heterogeneous storage (Phase 2) - Support APIs for
> using
> > storage tiers by the applications HDFS-5682
> >                 • Memory as storage tier HDFS-5851
> >         • YARN
> >                 • Dynamic Resource Configuration YARN-291
> >                 • NodeManager Restart YARN-1336
> >                 • ResourceManager HA Phase 2 YARN-556
> >                 • Support for admin-specified labels in YARN YARN-796
> >                 • Support for automatic, shared cache for YARN
> application
> > artifacts YARN-1492
> >                 • Support NodeGroup layer topology on YARN YARN-18
> >                 • Support for Docker containers in YARN YARN-1964
> >                 • YARN service registry YARN-913
> >
> >  My suspicion is, as is normal, some will make the cut and some won't.
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> > thanks,
> > Arun
> >
> >
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Andrew Wang <an...@cloudera.com>.

Hey Arun,

Maybe we could do a quick run through of the Roadmap wiki and add/retarget
things accordingly?

I think the KMS and transparent encryption are ready to go. We've got a
very few further bug fixes pending, but that's it.

Two HDFS things that I think probably won't make the end of the week are
archival storage (HDFS-6584) and single replica memory writes (HDFS-6581),
which I believe are under the HSM banner. HDFS-6484 was just merged to
trunk and I think needs a little more work before it goes into branch-2.
HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off
yet.

Just my 2c as I did not work directly on these features. I just generally
shy away from shipping bits quite this fresh.

Thanks,
Andrew

On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:

> Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>
> I'll create branch-2.6 by end of the week and we can go from there to
> stabilize it - hopefully in the next few weeks.
>
> Thoughts?
>
> thanks,
> Arun
>
> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
>
> > Folks,
> >
> >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >  Currently, here is the Roadmap per the wiki:
> >
> >         • HADOOP
> >                 • Credential provider HADOOP-10607
> >         • HDFS
> >                 • Heterogeneous storage (Phase 2) - Support APIs for
> using
> > storage tiers by the applications HDFS-5682
> >                 • Memory as storage tier HDFS-5851
> >         • YARN
> >                 • Dynamic Resource Configuration YARN-291
> >                 • NodeManager Restart YARN-1336
> >                 • ResourceManager HA Phase 2 YARN-556
> >                 • Support for admin-specified labels in YARN YARN-796
> >                 • Support for automatic, shared cache for YARN
> application
> > artifacts YARN-1492
> >                 • Support NodeGroup layer topology on YARN YARN-18
> >                 • Support for Docker containers in YARN YARN-1964
> >                 • YARN service registry YARN-913
> >
> >  My suspicion is, as is normal, some will make the cut and some won't.
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> > thanks,
> > Arun
> >
> >
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.

Thank for restarting this, Arun!

The following are the efforts that I am involved in directly or indirectly
and shepherding from YARN's point of view: rolling-upgrades (YARN-666),
work-preserving RM restart (YARN-556), minimal long-running services
support (YARN-896), including service-registry (YARN-913), timeline-service
stability/security (some sub-tasks of YARN-1935), node labels (YARN-296)
and some enhancements/bug fixes in cpu-scheduling/cgroups. We also have
reservations sub-system (YARN-1051) and  YARN shared cache (YARN-1492) that
I would like to get in and helping reviews with. Almost all these efforts
are on the edge of completion - some of them are just pending reviews.
Clearly some of them will be more stable than others, some of them make it
eventually, some may not.

+1 for branching it within a week so that we can stabilize it and let other
features go into branch-2, I am putting out all the stops to get the above
YARN efforts to be in reasonable shape for 2.6.

Regarding remaining items that I haven't paid much attention, NM restart
effort - mostly done by Jason Lowe - YARN-1336 is all in. YARN-291,
YARN-18, YARN-1964 are not going to cut from what I see.

I edited the road-map to reflect this latest reality.

+Vinod

On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:

> Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>
> I'll create branch-2.6 by end of the week and we can go from there to
> stabilize it - hopefully in the next few weeks.
>
> Thoughts?
>
> thanks,
> Arun
>
> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
>
> > Folks,
> >
> >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >  Currently, here is the Roadmap per the wiki:
> >
> >         • HADOOP
> >                 • Credential provider HADOOP-10607
> >         • HDFS
> >                 • Heterogeneous storage (Phase 2) - Support APIs for
> using
> > storage tiers by the applications HDFS-5682
> >                 • Memory as storage tier HDFS-5851
> >         • YARN
> >                 • Dynamic Resource Configuration YARN-291
> >                 • NodeManager Restart YARN-1336
> >                 • ResourceManager HA Phase 2 YARN-556
> >                 • Support for admin-specified labels in YARN YARN-796
> >                 • Support for automatic, shared cache for YARN
> application
> > artifacts YARN-1492
> >                 • Support NodeGroup layer topology on YARN YARN-18
> >                 • Support for Docker containers in YARN YARN-1964
> >                 • YARN service registry YARN-913
> >
> >  My suspicion is, as is normal, some will make the cut and some won't.
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> > thanks,
> > Arun
> >
> >
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.

Thank for restarting this, Arun!

The following are the efforts that I am involved in directly or indirectly
and shepherding from YARN's point of view: rolling-upgrades (YARN-666),
work-preserving RM restart (YARN-556), minimal long-running services
support (YARN-896), including service-registry (YARN-913), timeline-service
stability/security (some sub-tasks of YARN-1935), node labels (YARN-296)
and some enhancements/bug fixes in cpu-scheduling/cgroups. We also have
reservations sub-system (YARN-1051) and  YARN shared cache (YARN-1492) that
I would like to get in and helping reviews with. Almost all these efforts
are on the edge of completion - some of them are just pending reviews.
Clearly some of them will be more stable than others, some of them make it
eventually, some may not.

+1 for branching it within a week so that we can stabilize it and let other
features go into branch-2, I am putting out all the stops to get the above
YARN efforts to be in reasonable shape for 2.6.

Regarding remaining items that I haven't paid much attention, NM restart
effort - mostly done by Jason Lowe - YARN-1336 is all in. YARN-291,
YARN-18, YARN-1964 are not going to cut from what I see.

I edited the road-map to reflect this latest reality.

+Vinod

On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:

> Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>
> I'll create branch-2.6 by end of the week and we can go from there to
> stabilize it - hopefully in the next few weeks.
>
> Thoughts?
>
> thanks,
> Arun
>
> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
>
> > Folks,
> >
> >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >  Currently, here is the Roadmap per the wiki:
> >
> >         • HADOOP
> >                 • Credential provider HADOOP-10607
> >         • HDFS
> >                 • Heterogeneous storage (Phase 2) - Support APIs for
> using
> > storage tiers by the applications HDFS-5682
> >                 • Memory as storage tier HDFS-5851
> >         • YARN
> >                 • Dynamic Resource Configuration YARN-291
> >                 • NodeManager Restart YARN-1336
> >                 • ResourceManager HA Phase 2 YARN-556
> >                 • Support for admin-specified labels in YARN YARN-796
> >                 • Support for automatic, shared cache for YARN
> application
> > artifacts YARN-1492
> >                 • Support NodeGroup layer topology on YARN YARN-18
> >                 • Support for Docker containers in YARN YARN-1964
> >                 • YARN service registry YARN-913
> >
> >  My suspicion is, as is normal, some will make the cut and some won't.
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> > thanks,
> > Arun
> >
> >
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.

Thank for restarting this, Arun!

The following are the efforts that I am involved in directly or indirectly
and shepherding from YARN's point of view: rolling-upgrades (YARN-666),
work-preserving RM restart (YARN-556), minimal long-running services
support (YARN-896), including service-registry (YARN-913), timeline-service
stability/security (some sub-tasks of YARN-1935), node labels (YARN-296)
and some enhancements/bug fixes in cpu-scheduling/cgroups. We also have
reservations sub-system (YARN-1051) and  YARN shared cache (YARN-1492) that
I would like to get in and helping reviews with. Almost all these efforts
are on the edge of completion - some of them are just pending reviews.
Clearly some of them will be more stable than others, some of them make it
eventually, some may not.

+1 for branching it within a week so that we can stabilize it and let other
features go into branch-2, I am putting out all the stops to get the above
YARN efforts to be in reasonable shape for 2.6.

Regarding remaining items that I haven't paid much attention, NM restart
effort - mostly done by Jason Lowe - YARN-1336 is all in. YARN-291,
YARN-18, YARN-1964 are not going to cut from what I see.

I edited the road-map to reflect this latest reality.

+Vinod

On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com> wrote:

> Looks like most of the content is in and hadoop-2.6 is shaping up nicely.
>
> I'll create branch-2.6 by end of the week and we can go from there to
> stabilize it - hopefully in the next few weeks.
>
> Thoughts?
>
> thanks,
> Arun
>
> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
>
> > Folks,
> >
> >  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >  Currently, here is the Roadmap per the wiki:
> >
> >         • HADOOP
> >                 • Credential provider HADOOP-10607
> >         • HDFS
> >                 • Heterogeneous storage (Phase 2) - Support APIs for
> using
> > storage tiers by the applications HDFS-5682
> >                 • Memory as storage tier HDFS-5851
> >         • YARN
> >                 • Dynamic Resource Configuration YARN-291
> >                 • NodeManager Restart YARN-1336
> >                 • ResourceManager HA Phase 2 YARN-556
> >                 • Support for admin-specified labels in YARN YARN-796
> >                 • Support for automatic, shared cache for YARN
> application
> > artifacts YARN-1492
> >                 • Support NodeGroup layer topology on YARN YARN-18
> >                 • Support for Docker containers in YARN YARN-1964
> >                 • YARN service registry YARN-913
> >
> >  My suspicion is, as is normal, some will make the cut and some won't.
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> > thanks,
> > Arun
> >
> >
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun Murthy <ac...@hortonworks.com>.

Looks like most of the content is in and hadoop-2.6 is shaping up nicely.

I'll create branch-2.6 by end of the week and we can go from there to
stabilize it - hopefully in the next few weeks.

Thoughts?

thanks,
Arun

On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
>
>  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> hadoop-2.6.
>
>  Currently, here is the Roadmap per the wiki:
>
>         • HADOOP
>                 • Credential provider HADOOP-10607
>         • HDFS
>                 • Heterogeneous storage (Phase 2) - Support APIs for using
> storage tiers by the applications HDFS-5682
>                 • Memory as storage tier HDFS-5851
>         • YARN
>                 • Dynamic Resource Configuration YARN-291
>                 • NodeManager Restart YARN-1336
>                 • ResourceManager HA Phase 2 YARN-556
>                 • Support for admin-specified labels in YARN YARN-796
>                 • Support for automatic, shared cache for YARN application
> artifacts YARN-1492
>                 • Support NodeGroup layer topology on YARN YARN-18
>                 • Support for Docker containers in YARN YARN-1964
>                 • YARN service registry YARN-913
>
>  My suspicion is, as is normal, some will make the cut and some won't.
> Please do add/subtract from the list as appropriate. Ideally, it would be
> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence.
>
>  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> how they feel about this.
>
> thanks,
> Arun
>
>


-- 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun Murthy <ac...@hortonworks.com>.

Looks like most of the content is in and hadoop-2.6 is shaping up nicely.

I'll create branch-2.6 by end of the week and we can go from there to
stabilize it - hopefully in the next few weeks.

Thoughts?

thanks,
Arun

On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
>
>  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> hadoop-2.6.
>
>  Currently, here is the Roadmap per the wiki:
>
>         • HADOOP
>                 • Credential provider HADOOP-10607
>         • HDFS
>                 • Heterogeneous storage (Phase 2) - Support APIs for using
> storage tiers by the applications HDFS-5682
>                 • Memory as storage tier HDFS-5851
>         • YARN
>                 • Dynamic Resource Configuration YARN-291
>                 • NodeManager Restart YARN-1336
>                 • ResourceManager HA Phase 2 YARN-556
>                 • Support for admin-specified labels in YARN YARN-796
>                 • Support for automatic, shared cache for YARN application
> artifacts YARN-1492
>                 • Support NodeGroup layer topology on YARN YARN-18
>                 • Support for Docker containers in YARN YARN-1964
>                 • YARN service registry YARN-913
>
>  My suspicion is, as is normal, some will make the cut and some won't.
> Please do add/subtract from the list as appropriate. Ideally, it would be
> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence.
>
>  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> how they feel about this.
>
> thanks,
> Arun
>
>


-- 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun Murthy <ac...@hortonworks.com>.

Looks like most of the content is in and hadoop-2.6 is shaping up nicely.

I'll create branch-2.6 by end of the week and we can go from there to
stabilize it - hopefully in the next few weeks.

Thoughts?

thanks,
Arun

On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
>
>  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> hadoop-2.6.
>
>  Currently, here is the Roadmap per the wiki:
>
>         • HADOOP
>                 • Credential provider HADOOP-10607
>         • HDFS
>                 • Heterogeneous storage (Phase 2) - Support APIs for using
> storage tiers by the applications HDFS-5682
>                 • Memory as storage tier HDFS-5851
>         • YARN
>                 • Dynamic Resource Configuration YARN-291
>                 • NodeManager Restart YARN-1336
>                 • ResourceManager HA Phase 2 YARN-556
>                 • Support for admin-specified labels in YARN YARN-796
>                 • Support for automatic, shared cache for YARN application
> artifacts YARN-1492
>                 • Support NodeGroup layer topology on YARN YARN-18
>                 • Support for Docker containers in YARN YARN-1964
>                 • YARN service registry YARN-913
>
>  My suspicion is, as is normal, some will make the cut and some won't.
> Please do add/subtract from the list as appropriate. Ideally, it would be
> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence.
>
>  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> how they feel about this.
>
> thanks,
> Arun
>
>


-- 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.6

Posted by Arun Murthy <ac...@hortonworks.com>.

Looks like most of the content is in and hadoop-2.6 is shaping up nicely.

I'll create branch-2.6 by end of the week and we can go from there to
stabilize it - hopefully in the next few weeks.

Thoughts?

thanks,
Arun

On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
>
>  With hadoop-2.5 nearly done, it's time to start thinking ahead to
> hadoop-2.6.
>
>  Currently, here is the Roadmap per the wiki:
>
>         • HADOOP
>                 • Credential provider HADOOP-10607
>         • HDFS
>                 • Heterogeneous storage (Phase 2) - Support APIs for using
> storage tiers by the applications HDFS-5682
>                 • Memory as storage tier HDFS-5851
>         • YARN
>                 • Dynamic Resource Configuration YARN-291
>                 • NodeManager Restart YARN-1336
>                 • ResourceManager HA Phase 2 YARN-556
>                 • Support for admin-specified labels in YARN YARN-796
>                 • Support for automatic, shared cache for YARN application
> artifacts YARN-1492
>                 • Support NodeGroup layer topology on YARN YARN-18
>                 • Support for Docker containers in YARN YARN-1964
>                 • YARN service registry YARN-913
>
>  My suspicion is, as is normal, some will make the cut and some won't.
> Please do add/subtract from the list as appropriate. Ideally, it would be
> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence.
>
>  More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> how they feel about this.
>
> thanks,
> Arun
>
>


-- 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.