You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Arun C Murthy <ac...@hortonworks.com> on 2014/10/03 02:02:11 UTC
Re: Thinking ahead to hadoop-2.6
Re-sending… doesn't look like it made through.
On Sep 30, 2014, at 10:42 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Folks,
>
> I've created branch-2.6 to stabilize the release.
>
> Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already.
>
> By default new features should now be targeted to the version "2.7" henceforth - I've ensure all the projects have that version on jira.
>
> thanks,
> Arun
>
> On Sep 26, 2014, at 1:08 AM, Arun Murthy <ac...@hortonworks.com> wrote:
>
>> Sounds good. I'll branch this weekend and we can merge the jiras we
>> discussed in this thread as they they get wrapped next week.
>>
>> Thanks everyone.
>>
>> Arun
>>
>>
>>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> We can branch off in a week or two so that work on branch-2 itself can go
>>> ahead with other features that can't fit in 2.6. Independent of that, we
>>> can then decide on the timeline of the release candidates once branch-2.6
>>> is close to being done w.r.t the planned features.
>>>
>>> Branching it off can let us focus on specific features that we want in for
>>> 2.6 and then eventually blockers for the release, nothing else. There is a
>>> trivial pain of committing to one more branch, but it's worth it in this
>>> case IMO.
>>>
>>> A lot of efforts are happening in parallel from the YARN side from where I
>>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we
>>> don't branch off and selectively try to get stuff in, it is likely to be in
>>> a perpetual delay.
>>>
>>> My 2 cents.
>>>
>>> +Vinod
>>>
>>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <su...@hortonworks.com>
>>> wrote:
>>>
>>>> Given some of the features are in final stages of stabilization,
>>>> Arun, we should hold off creating 2.6 branch or building an RC by a week?
>>>> All the features in flux are important ones and worth delaying the release
>>>> by a week.
>>>>
>>>> On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>>
>>>>> Hey Nicholas,
>>>>>
>>>>> My concern about Archival Storage isn't related to the code quality or
>>>> the
>>>>> size of the feature. I think that you and Jing did good work. My concern
>>>> is
>>>>> that once we ship, we're locked into that set of archival storage APIs,
>>>> and
>>>>> these APIs are not yet finalized. Simply being able to turn off the
>>>> feature
>>>>> does not change the compatibility story.
>>>>>
>>>>> I'm willing to devote time to help review these JIRAs and kick the tires
>>>> on
>>>>> the APIs, but my point above was that I'm not sure it'd all be done by
>>>> the
>>>>> end of the week. Testing might also reveal additional changes that need
>>>> to
>>>>> be made, which also might not happen by end-of-week.
>>>>>
>>>>> I guess the question before us is if we're comfortable putting something
>>>> in
>>>>> branch-2.6 and then potentially adding API changes after. I'm okay with
>>>>> that as long as we're all aware that this might happen.
>>>>>
>>>>> Arun, as RM is this cool with you? Again, I like this feature and I'm
>>>> fine
>>>>> with it's inclusion, just a heads up that we might need some extra time
>>>> to
>>>>> finalize things before an RC can be cut.
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>> On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>>>>> s29752-hadoopdev@yahoo.com.invalid> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am worry about KMS and transparent encryption since there are quite
>>>>> many
>>>>>> bugs discovered after it got merged to branch-2. It gives us an
>>>>> impression
>>>>>> that the feature is not yet well tested. Indeed, transparent
>>>> encryption
>>>>> is
>>>>>> a complicated feature which changes the core part of HDFS. It is not
>>>>> easy
>>>>>> to get everything right.
>>>>>>
>>>>>>
>>>>>> For HDFS-6584: Archival Storage, it is a relatively simple and low risk
>>>>>> feature. It introduces a new storage type ARCHIVE and the concept of
>>>>> block
>>>>>> storage policy to HDFS. When a cluster is configured with ARCHIVE
>>>>> storage,
>>>>>> the blocks will be stored using the appropriate storage types specified
>>>>> by
>>>>>> storage policies assigned to the files/directories. Cluster admin
>>>> could
>>>>>> disable the feature by simply not configuring any storage type and not
>>>>>> setting any storage policy as before. As Suresh mentioned, HDFS-6584
>>>> is
>>>>>> in the final stages to be merged to branch-2.
>>>>>>
>>>>>> Regards,
>>>>>> Tsz-Wo
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas <
>>>>>> suresh@hortonworks.com> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I actually would like to see both archival storage and single replica
>>>>>>> memory writes to be in 2.6 release. Archival storage is in the final
>>>>>> stages
>>>>>>> of getting ready for branch-2 merge as Nicholas has already indicated
>>>> on
>>>>>>> the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
>>>>> these
>>>>>>> features are being in development for sometime.
>>>>>>>
>>>>>>> On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <
>>>> andrew.wang@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey Arun,
>>>>>>>>
>>>>>>>> Maybe we could do a quick run through of the Roadmap wiki and
>>>>>> add/retarget
>>>>>>>> things accordingly?
>>>>>>>>
>>>>>>>> I think the KMS and transparent encryption are ready to go. We've
>>>> got
>>>>> a
>>>>>>>> very few further bug fixes pending, but that's it.
>>>>>>>>
>>>>>>>> Two HDFS things that I think probably won't make the end of the week
>>>>> are
>>>>>>>> archival storage (HDFS-6584) and single replica memory writes
>>>>>> (HDFS-6581),
>>>>>>>> which I believe are under the HSM banner. HDFS-6484 was just merged
>>>> to
>>>>>>>> trunk and I think needs a little more work before it goes into
>>>>> branch-2.
>>>>>>>> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
>>>> further
>>>>>> off
>>>>>>>> yet.
>>>>>>>>
>>>>>>>> Just my 2c as I did not work directly on these features. I just
>>>>>> generally
>>>>>>>> shy away from shipping bits quite this fresh.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Looks like most of the content is in and hadoop-2.6 is shaping up
>>>>>> nicely.
>>>>>>>>>
>>>>>>>>> I'll create branch-2.6 by end of the week and we can go from there
>>>>> to
>>>>>>>>> stabilize it - hopefully in the next few weeks.
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <
>>>> acm@hortonworks.com
>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Folks,
>>>>>>>>>>
>>>>>>>>>> With hadoop-2.5 nearly done, it's time to start thinking ahead
>>>> to
>>>>>>>>>> hadoop-2.6.
>>>>>>>>>>
>>>>>>>>>> Currently, here is the Roadmap per the wiki:
>>>>>>>>>>
>>>>>>>>>> • HADOOP
>>>>>>>>>> • Credential provider HADOOP-10607
>>>>>>>>>> • HDFS
>>>>>>>>>> • Heterogeneous storage (Phase 2) - Support APIs
>>>>> for
>>>>>>>>> using
>>>>>>>>>> storage tiers by the applications HDFS-5682
>>>>>>>>>> • Memory as storage tier HDFS-5851
>>>>>>>>>> • YARN
>>>>>>>>>> • Dynamic Resource Configuration YARN-291
>>>>>>>>>> • NodeManager Restart YARN-1336
>>>>>>>>>> • ResourceManager HA Phase 2 YARN-556
>>>>>>>>>> • Support for admin-specified labels in YARN
>>>>>> YARN-796
>>>>>>>>>> • Support for automatic, shared cache for YARN
>>>>>>>>> application
>>>>>>>>>> artifacts YARN-1492
>>>>>>>>>> • Support NodeGroup layer topology on YARN
>>>> YARN-18
>>>>>>>>>> • Support for Docker containers in YARN
>>>> YARN-1964
>>>>>>>>>> • YARN service registry YARN-913
>>>>>>>>>>
>>>>>>>>>> My suspicion is, as is normal, some will make the cut and some
>>>>>> won't.
>>>>>>>>>> Please do add/subtract from the list as appropriate. Ideally, it
>>>>>> would
>>>>>>>> be
>>>>>>>>>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep
>>>> up a
>>>>>>>>> cadence.
>>>>>>>>>>
>>>>>>>>>> More importantly, as we discussed previously, we'd like
>>>>> hadoop-2.6
>>>>>> to
>>>>>>>> be
>>>>>>>>>> the *last* Apache Hadoop 2.x release which support JDK6. I'll
>>>>> start
>>>>>> a
>>>>>>>>>> discussion with other communities (HBase, Pig, Hive, Oozie etc.)
>>>>> and
>>>>>>>> see
>>>>>>>>>> how they feel about this.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity
>>>>>>>> to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>> --
>>>> http://hortonworks.com/download/
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/hdp/
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.