You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Allen Wittenauer <aw...@altiscale.com> on 2014/09/15 19:48:18 UTC
In hindsight... Re: Thinking ahead to hadoop-2.6
It’s now September. With the passage of time, I have a lot of doubts about this plan and where that trajectory takes us.
* The list of changes that are already in branch-2 scare the crap out of any risk adverse person (Hello to my fellow operations people!). Not only are the number of changes extremely high, but in addition there are a lot of major, blockbuster features in what is supposed to be a minor release. Combined with the fact that we’ve had to do some micro releases, it seems to hint that branch-2 is getting less stable over time.
* One of the plans talked about was rolling a 2.7 release that drops JDK6 and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too late for us to talk about JDK7 and we need to start thinking about JDK8.
* trunk is currently sitting at 3 years old. There is a lot of stuff that has been hanging around that really needs to get into people hands so that we can start stabilizing it for a “real” release.
To me this all says one thing:
Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all sustaining work. This gives the rest of the community time to move to JDK8 if they haven’t already. For downstream vendors, it gives a roadmap for their customers who will be asking about JDK8 sooner rather than later. By the time 3.0 stabilizes, we’re probably looking at April, which is perfect timing.
One of the issues I’ve heard mention is that 3.0 doesn’t have anything “compelling” in it. Well, dropping 2.6 makes the feature list the carrot, JDK8 support is obviously the stick.
Thoughts?
On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan <su...@apache.org> wrote:
> Thanks for initiating the thread Arun.
>
> Can we add YARN-1051 <https://issues.apache.org/jira/browse/YARN-1051> to
> the list? We have most of the patches for the sub-JIRAs under review and
> have committed a couple.
>
> -Subru
>
> ---------- Forwarded message ----------
>
> From: Arun C Murthy <ac...@hortonworks.com>
>
> Date: Tue, Aug 12, 2014 at 1:34 PM
>
> Subject: Thinking ahead to hadoop-2.6
>
> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>
> "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
>
>
>
>
>
> Folks,
>
>
>
> With hadoop-2.5 nearly done, it's time to start thinking ahead to
> hadoop-2.6.
>
>
>
> Currently, here is the Roadmap per the wiki:
>
>
>
> • HADOOP
>
> • Credential provider HADOOP-10607
>
> • HDFS
>
> • Heterogeneous storage (Phase 2) - Support APIs for using
> storage tiers by the applications HDFS-5682
>
> • Memory as storage tier HDFS-5851
>
> • YARN
>
> • Dynamic Resource Configuration YARN-291
>
> • NodeManager Restart YARN-1336
>
> • ResourceManager HA Phase 2 YARN-556
>
> • Support for admin-specified labels in YARN YARN-796
>
> • Support for automatic, shared cache for YARN application
> artifacts YARN-1492
>
> • Support NodeGroup layer topology on YARN YARN-18
>
> • Support for Docker containers in YARN YARN-1964
>
> • YARN service registry YARN-913
>
>
>
> My suspicion is, as is normal, some will make the cut and some won't.
>
> Please do add/subtract from the list as appropriate. Ideally, it would be
> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence.
>
>
>
> More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> how they feel about this.
>
>
>
> thanks,
>
> Arun
>
>
>
>
>
> --
>
> CONFIDENTIALITY NOTICE
>
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Sep 15, 2014, at 11:17 AM, Colin McCabe <cm...@alumni.cmu.edu> wrote:
> On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>>
>> It’s now September. With the passage of time, I have a lot of doubts about this plan and where that trajectory takes us.
>>
>> * The list of changes that are already in branch-2 scare the crap out of any risk adverse person (Hello to my fellow operations people!). Not only are the number of changes extremely high, but in addition there are a lot of major, blockbuster features in what is supposed to be a minor release. Combined with the fact that we’ve had to do some micro releases, it seems to hint that branch-2 is getting less stable over time.
>
> I don't see what is so scary about 2.6, can you be more concrete? It
> seems like a pretty normal release to me and most of the new features
> are optional.
>
> I also don't see why you think that "branch-2 is getting less stable
> over time." Actually, I think that branch-2 has gotten more stable
> over time as people have finally gotten around to upgrading from 1.x
> or earlier, and contributed their efforts to addressing regressions in
> branch-2.
>
Please re-read what I wrote above.
>> * One of the plans talked about was rolling a 2.7 release that drops JDK6 and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too late for us to talk about JDK7 and we need to start thinking about JDK8.
>>
>> * trunk is currently sitting at 3 years old. There is a lot of stuff that has been hanging around that really needs to get into people hands so that we can start stabilizing it for a “real” release.
>
> We have been pretty careful about minimizing trunk's divergence from
> branch-2. I can't think of an example of anything in trunk that
> "really needs to get into people's hands"-- did I forget something?
There isn't anything in *any* of the recent branch-2 releases that qualifies as "really needs to get into people's hands" either, so I'm not sure what your point here is.
>> To me this all says one thing:
>>
>> Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all sustaining work. This gives the rest of the community time to move to JDK8 if they haven’t already. For downstream vendors, it gives a roadmap for their customers who will be asking about JDK8 sooner rather than later. By the time 3.0 stabilizes, we’re probably looking at April, which is perfect timing.
>>
>> One of the issues I’ve heard mention is that 3.0 doesn’t have anything “compelling” in it. Well, dropping 2.6 makes the feature list the carrot, JDK8 support is obviously the stick.
>>
>> Thoughts?
>
> As we've discussed before, supporting JDK8 is very different from
> forcing people to use JDK8. branch-2 and Hadoop 2.6 most certainly
> should support JDK8, and most certainly NOT force people to use JDK8.
We aren't the ones forcing the JDK8 issue: Oracle is. Users who want a viable, supported JDK will be moving to 8 by April.
> Cloudera has been using JDK7 internally for a long time, and
> recommending it to customers too.
Thank you for using your vendor hat to support my point: JDK7 works fine with Hadoop 2 now, so those users who can't upgrade to JDK8 for whatever reason still have a viable option if they want to use Hadoop. After all, again as you point out above, all of the new bits are optional features and could get pushed to a later release. We can continue to make security and critical bug fixes against 2.5.x and start pushing those optional features into a newly viable 3.x release. It also supports our major.minor.micro nomenclature as well as puts us in line with a lot of other Apache projects. (as pointed out to me today: http://t.co/Ry8Zu6yZkd )
> Some developers are using JDK8 as
> well. It works fine (although I'm sure there will be bugs and
> workarounds that get reported and fixed as more people migrate).
All of the people that I know that are doing JDK8 are applying patches to make it work. (See HADOOP-11090)
> I don't see this particular issue as a reason to change the schedule.
Look at this from a long-term perspective. If we make the move to support JDK7 as the base line in the next few months, we're essentially saying that we're willing to keep JDK7 working for the next 3-4 years (given our normal JVM update process). This isn't a viable strategy given that Oracle (you know, the people that provide the JVM) is about to drop support/provide updates in 7 months and JDK9 in early access with an expected ship date in 2016. JDK7 will be ancient at that point, even worse that JDK6 feels now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Sep 15, 2014, at 11:17 AM, Colin McCabe <cm...@alumni.cmu.edu> wrote:
> On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>>
>> It’s now September. With the passage of time, I have a lot of doubts about this plan and where that trajectory takes us.
>>
>> * The list of changes that are already in branch-2 scare the crap out of any risk adverse person (Hello to my fellow operations people!). Not only are the number of changes extremely high, but in addition there are a lot of major, blockbuster features in what is supposed to be a minor release. Combined with the fact that we’ve had to do some micro releases, it seems to hint that branch-2 is getting less stable over time.
>
> I don't see what is so scary about 2.6, can you be more concrete? It
> seems like a pretty normal release to me and most of the new features
> are optional.
>
> I also don't see why you think that "branch-2 is getting less stable
> over time." Actually, I think that branch-2 has gotten more stable
> over time as people have finally gotten around to upgrading from 1.x
> or earlier, and contributed their efforts to addressing regressions in
> branch-2.
>
Please re-read what I wrote above.
>> * One of the plans talked about was rolling a 2.7 release that drops JDK6 and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too late for us to talk about JDK7 and we need to start thinking about JDK8.
>>
>> * trunk is currently sitting at 3 years old. There is a lot of stuff that has been hanging around that really needs to get into people hands so that we can start stabilizing it for a “real” release.
>
> We have been pretty careful about minimizing trunk's divergence from
> branch-2. I can't think of an example of anything in trunk that
> "really needs to get into people's hands"-- did I forget something?
There isn't anything in *any* of the recent branch-2 releases that qualifies as "really needs to get into people's hands" either, so I'm not sure what your point here is.
>> To me this all says one thing:
>>
>> Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all sustaining work. This gives the rest of the community time to move to JDK8 if they haven’t already. For downstream vendors, it gives a roadmap for their customers who will be asking about JDK8 sooner rather than later. By the time 3.0 stabilizes, we’re probably looking at April, which is perfect timing.
>>
>> One of the issues I’ve heard mention is that 3.0 doesn’t have anything “compelling” in it. Well, dropping 2.6 makes the feature list the carrot, JDK8 support is obviously the stick.
>>
>> Thoughts?
>
> As we've discussed before, supporting JDK8 is very different from
> forcing people to use JDK8. branch-2 and Hadoop 2.6 most certainly
> should support JDK8, and most certainly NOT force people to use JDK8.
We aren't the ones forcing the JDK8 issue: Oracle is. Users who want a viable, supported JDK will be moving to 8 by April.
> Cloudera has been using JDK7 internally for a long time, and
> recommending it to customers too.
Thank you for using your vendor hat to support my point: JDK7 works fine with Hadoop 2 now, so those users who can't upgrade to JDK8 for whatever reason still have a viable option if they want to use Hadoop. After all, again as you point out above, all of the new bits are optional features and could get pushed to a later release. We can continue to make security and critical bug fixes against 2.5.x and start pushing those optional features into a newly viable 3.x release. It also supports our major.minor.micro nomenclature as well as puts us in line with a lot of other Apache projects. (as pointed out to me today: http://t.co/Ry8Zu6yZkd )
> Some developers are using JDK8 as
> well. It works fine (although I'm sure there will be bugs and
> workarounds that get reported and fixed as more people migrate).
All of the people that I know that are doing JDK8 are applying patches to make it work. (See HADOOP-11090)
> I don't see this particular issue as a reason to change the schedule.
Look at this from a long-term perspective. If we make the move to support JDK7 as the base line in the next few months, we're essentially saying that we're willing to keep JDK7 working for the next 3-4 years (given our normal JVM update process). This isn't a viable strategy given that Oracle (you know, the people that provide the JVM) is about to drop support/provide updates in 7 months and JDK9 in early access with an expected ship date in 2016. JDK7 will be ancient at that point, even worse that JDK6 feels now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Sep 15, 2014, at 11:17 AM, Colin McCabe <cm...@alumni.cmu.edu> wrote:
> On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>>
>> It’s now September. With the passage of time, I have a lot of doubts about this plan and where that trajectory takes us.
>>
>> * The list of changes that are already in branch-2 scare the crap out of any risk adverse person (Hello to my fellow operations people!). Not only are the number of changes extremely high, but in addition there are a lot of major, blockbuster features in what is supposed to be a minor release. Combined with the fact that we’ve had to do some micro releases, it seems to hint that branch-2 is getting less stable over time.
>
> I don't see what is so scary about 2.6, can you be more concrete? It
> seems like a pretty normal release to me and most of the new features
> are optional.
>
> I also don't see why you think that "branch-2 is getting less stable
> over time." Actually, I think that branch-2 has gotten more stable
> over time as people have finally gotten around to upgrading from 1.x
> or earlier, and contributed their efforts to addressing regressions in
> branch-2.
>
Please re-read what I wrote above.
>> * One of the plans talked about was rolling a 2.7 release that drops JDK6 and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too late for us to talk about JDK7 and we need to start thinking about JDK8.
>>
>> * trunk is currently sitting at 3 years old. There is a lot of stuff that has been hanging around that really needs to get into people hands so that we can start stabilizing it for a “real” release.
>
> We have been pretty careful about minimizing trunk's divergence from
> branch-2. I can't think of an example of anything in trunk that
> "really needs to get into people's hands"-- did I forget something?
There isn't anything in *any* of the recent branch-2 releases that qualifies as "really needs to get into people's hands" either, so I'm not sure what your point here is.
>> To me this all says one thing:
>>
>> Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all sustaining work. This gives the rest of the community time to move to JDK8 if they haven’t already. For downstream vendors, it gives a roadmap for their customers who will be asking about JDK8 sooner rather than later. By the time 3.0 stabilizes, we’re probably looking at April, which is perfect timing.
>>
>> One of the issues I’ve heard mention is that 3.0 doesn’t have anything “compelling” in it. Well, dropping 2.6 makes the feature list the carrot, JDK8 support is obviously the stick.
>>
>> Thoughts?
>
> As we've discussed before, supporting JDK8 is very different from
> forcing people to use JDK8. branch-2 and Hadoop 2.6 most certainly
> should support JDK8, and most certainly NOT force people to use JDK8.
We aren't the ones forcing the JDK8 issue: Oracle is. Users who want a viable, supported JDK will be moving to 8 by April.
> Cloudera has been using JDK7 internally for a long time, and
> recommending it to customers too.
Thank you for using your vendor hat to support my point: JDK7 works fine with Hadoop 2 now, so those users who can't upgrade to JDK8 for whatever reason still have a viable option if they want to use Hadoop. After all, again as you point out above, all of the new bits are optional features and could get pushed to a later release. We can continue to make security and critical bug fixes against 2.5.x and start pushing those optional features into a newly viable 3.x release. It also supports our major.minor.micro nomenclature as well as puts us in line with a lot of other Apache projects. (as pointed out to me today: http://t.co/Ry8Zu6yZkd )
> Some developers are using JDK8 as
> well. It works fine (although I'm sure there will be bugs and
> workarounds that get reported and fixed as more people migrate).
All of the people that I know that are doing JDK8 are applying patches to make it work. (See HADOOP-11090)
> I don't see this particular issue as a reason to change the schedule.
Look at this from a long-term perspective. If we make the move to support JDK7 as the base line in the next few months, we're essentially saying that we're willing to keep JDK7 working for the next 3-4 years (given our normal JVM update process). This isn't a viable strategy given that Oracle (you know, the people that provide the JVM) is about to drop support/provide updates in 7 months and JDK9 in early access with an expected ship date in 2016. JDK7 will be ancient at that point, even worse that JDK6 feels now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Sep 15, 2014, at 11:17 AM, Colin McCabe <cm...@alumni.cmu.edu> wrote:
> On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>>
>> It’s now September. With the passage of time, I have a lot of doubts about this plan and where that trajectory takes us.
>>
>> * The list of changes that are already in branch-2 scare the crap out of any risk adverse person (Hello to my fellow operations people!). Not only are the number of changes extremely high, but in addition there are a lot of major, blockbuster features in what is supposed to be a minor release. Combined with the fact that we’ve had to do some micro releases, it seems to hint that branch-2 is getting less stable over time.
>
> I don't see what is so scary about 2.6, can you be more concrete? It
> seems like a pretty normal release to me and most of the new features
> are optional.
>
> I also don't see why you think that "branch-2 is getting less stable
> over time." Actually, I think that branch-2 has gotten more stable
> over time as people have finally gotten around to upgrading from 1.x
> or earlier, and contributed their efforts to addressing regressions in
> branch-2.
>
Please re-read what I wrote above.
>> * One of the plans talked about was rolling a 2.7 release that drops JDK6 and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too late for us to talk about JDK7 and we need to start thinking about JDK8.
>>
>> * trunk is currently sitting at 3 years old. There is a lot of stuff that has been hanging around that really needs to get into people hands so that we can start stabilizing it for a “real” release.
>
> We have been pretty careful about minimizing trunk's divergence from
> branch-2. I can't think of an example of anything in trunk that
> "really needs to get into people's hands"-- did I forget something?
There isn't anything in *any* of the recent branch-2 releases that qualifies as "really needs to get into people's hands" either, so I'm not sure what your point here is.
>> To me this all says one thing:
>>
>> Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all sustaining work. This gives the rest of the community time to move to JDK8 if they haven’t already. For downstream vendors, it gives a roadmap for their customers who will be asking about JDK8 sooner rather than later. By the time 3.0 stabilizes, we’re probably looking at April, which is perfect timing.
>>
>> One of the issues I’ve heard mention is that 3.0 doesn’t have anything “compelling” in it. Well, dropping 2.6 makes the feature list the carrot, JDK8 support is obviously the stick.
>>
>> Thoughts?
>
> As we've discussed before, supporting JDK8 is very different from
> forcing people to use JDK8. branch-2 and Hadoop 2.6 most certainly
> should support JDK8, and most certainly NOT force people to use JDK8.
We aren't the ones forcing the JDK8 issue: Oracle is. Users who want a viable, supported JDK will be moving to 8 by April.
> Cloudera has been using JDK7 internally for a long time, and
> recommending it to customers too.
Thank you for using your vendor hat to support my point: JDK7 works fine with Hadoop 2 now, so those users who can't upgrade to JDK8 for whatever reason still have a viable option if they want to use Hadoop. After all, again as you point out above, all of the new bits are optional features and could get pushed to a later release. We can continue to make security and critical bug fixes against 2.5.x and start pushing those optional features into a newly viable 3.x release. It also supports our major.minor.micro nomenclature as well as puts us in line with a lot of other Apache projects. (as pointed out to me today: http://t.co/Ry8Zu6yZkd )
> Some developers are using JDK8 as
> well. It works fine (although I'm sure there will be bugs and
> workarounds that get reported and fixed as more people migrate).
All of the people that I know that are doing JDK8 are applying patches to make it work. (See HADOOP-11090)
> I don't see this particular issue as a reason to change the schedule.
Look at this from a long-term perspective. If we make the move to support JDK7 as the base line in the next few months, we're essentially saying that we're willing to keep JDK7 working for the next 3-4 years (given our normal JVM update process). This isn't a viable strategy given that Oracle (you know, the people that provide the JVM) is about to drop support/provide updates in 7 months and JDK9 in early access with an expected ship date in 2016. JDK7 will be ancient at that point, even worse that JDK6 feels now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> It’s now September. With the passage of time, I have a lot of doubts about this plan and where that trajectory takes us.
>
> * The list of changes that are already in branch-2 scare the crap out of any risk adverse person (Hello to my fellow operations people!). Not only are the number of changes extremely high, but in addition there are a lot of major, blockbuster features in what is supposed to be a minor release. Combined with the fact that we’ve had to do some micro releases, it seems to hint that branch-2 is getting less stable over time.
I don't see what is so scary about 2.6, can you be more concrete? It
seems like a pretty normal release to me and most of the new features
are optional.
I also don't see why you think that "branch-2 is getting less stable
over time." Actually, I think that branch-2 has gotten more stable
over time as people have finally gotten around to upgrading from 1.x
or earlier, and contributed their efforts to addressing regressions in
branch-2.
> * One of the plans talked about was rolling a 2.7 release that drops JDK6 and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too late for us to talk about JDK7 and we need to start thinking about JDK8.
>
> * trunk is currently sitting at 3 years old. There is a lot of stuff that has been hanging around that really needs to get into people hands so that we can start stabilizing it for a “real” release.
We have been pretty careful about minimizing trunk's divergence from
branch-2. I can't think of an example of anything in trunk that
"really needs to get into people's hands"-- did I forget something?
>
>
> To me this all says one thing:
>
> Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all sustaining work. This gives the rest of the community time to move to JDK8 if they haven’t already. For downstream vendors, it gives a roadmap for their customers who will be asking about JDK8 sooner rather than later. By the time 3.0 stabilizes, we’re probably looking at April, which is perfect timing.
>
> One of the issues I’ve heard mention is that 3.0 doesn’t have anything “compelling” in it. Well, dropping 2.6 makes the feature list the carrot, JDK8 support is obviously the stick.
>
> Thoughts?
As we've discussed before, supporting JDK8 is very different from
forcing people to use JDK8. branch-2 and Hadoop 2.6 most certainly
should support JDK8, and most certainly NOT force people to use JDK8.
Cloudera has been using JDK7 internally for a long time, and
recommending it to customers too. Some developers are using JDK8 as
well. It works fine (although I'm sure there will be bugs and
workarounds that get reported and fixed as more people migrate). I
don't see this particular issue as a reason to change the schedule.
best,
Colin
>
>
>
>
> On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan <su...@apache.org> wrote:
>
>> Thanks for initiating the thread Arun.
>>
>> Can we add YARN-1051 <https://issues.apache.org/jira/browse/YARN-1051> to
>> the list? We have most of the patches for the sub-JIRAs under review and
>> have committed a couple.
>>
>> -Subru
>>
>> ---------- Forwarded message ----------
>>
>> From: Arun C Murthy <ac...@hortonworks.com>
>>
>> Date: Tue, Aug 12, 2014 at 1:34 PM
>>
>> Subject: Thinking ahead to hadoop-2.6
>>
>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>> hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>>
>> "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
>>
>>
>>
>>
>>
>> Folks,
>>
>>
>>
>> With hadoop-2.5 nearly done, it's time to start thinking ahead to
>> hadoop-2.6.
>>
>>
>>
>> Currently, here is the Roadmap per the wiki:
>>
>>
>>
>> • HADOOP
>>
>> • Credential provider HADOOP-10607
>>
>> • HDFS
>>
>> • Heterogeneous storage (Phase 2) - Support APIs for using
>> storage tiers by the applications HDFS-5682
>>
>> • Memory as storage tier HDFS-5851
>>
>> • YARN
>>
>> • Dynamic Resource Configuration YARN-291
>>
>> • NodeManager Restart YARN-1336
>>
>> • ResourceManager HA Phase 2 YARN-556
>>
>> • Support for admin-specified labels in YARN YARN-796
>>
>> • Support for automatic, shared cache for YARN application
>> artifacts YARN-1492
>>
>> • Support NodeGroup layer topology on YARN YARN-18
>>
>> • Support for Docker containers in YARN YARN-1964
>>
>> • YARN service registry YARN-913
>>
>>
>>
>> My suspicion is, as is normal, some will make the cut and some won't.
>>
>> Please do add/subtract from the list as appropriate. Ideally, it would be
>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence.
>>
>>
>>
>> More importantly, as we discussed previously, we'd like hadoop-2.6 to be
>> the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
>> discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
>> how they feel about this.
>>
>>
>>
>> thanks,
>>
>> Arun
>>
>>
>>
>>
>>
>> --
>>
>> CONFIDENTIALITY NOTICE
>>
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Arun Murthy <ac...@hortonworks.com>.
Sorry, coming to discussion late.
We all agreed that 2.6 would the *last* release supporting JDK6 and
hadoop-2.7 would drop support for JDK6. We could easily do 2.7 right after
2.6 (maybe with few critical bug-fixes) with the defining feature of 2.7
being *JDK7 only*. I've checked with HBase, Pig and other communities and
they are good with this. I'm thinking I'll follow up 2-3 wks after 2.6 goes
out to release 2.7 which drops JDK6.
We can certainly add support for JDK8 as early as 2.7 if there are
volunteers - clearly we won't make it depend on JDK8 features right away;
since it would still need to support JDK7.
To recap: hadoop-2.7 onwards would be minimum JDK7, with potential support
for JDK8. We can revisit our discussion from a few months ago to discuss
when we *drop* support for JDK7; clearly something I'd like to avoid doing
in haste.
thanks,
Arun
On Thu, Sep 18, 2014 at 8:41 AM, Alejandro Abdelnur <tu...@gmail.com>
wrote:
> Am I missing something, or we already agreed that after 2.5 release we
> would move trunk and branch-2 to java 7?
>
> On Wed, Sep 17, 2014 at 3:33 PM, Travis Thompson <
> travis.r.thompson@gmail.com> wrote:
>
>> There's actually an umbrella JIRA to track issues with JDK8
>> (HADOOP-11090), in case anyone missed it.
>>
>> At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for
>> about a month now with some mixed results. It definitely works but
>> there are issues, mostly around virtual memory exploding. The reason
>> we took the jump early is there is a company wide push to move to JDK8
>> ASAP, I suspect this isn't something unique to LinkedIn. To get this
>> to work with security enabled, we've had to apply patches not even in
>> trunk yet because they break JDK6 compatibility.
>>
>> From my perspective, based on what I've seen and people I've talked
>> to, there is a huge push to move to JDK8 ASAP so it's becoming
>> increasingly urgent to at least get support to run on JDK8.
>>
>> On Wed, Sep 17, 2014 at 9:55 AM, Allen Wittenauer <aw...@altiscale.com>
>> wrote:
>> >
>> > On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>> >>
>> >> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down
>> the
>> >> filesystem binding with more tests than ever before.
>> >
>> > FWIW, based upon my survey of JIRA, there are a lot of unit
>> test fixes that are only in trunk.
>> >
>> >> But I am also aware of large organisations that are still on Java 6.
>> >> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can
>> help
>> >> them plan.
>> >
>> > Planning is a big thing. That’s one of the reasons why it’d be
>> prudent to start doing 3.0+JDK8 now as well. Even if April slips, other
>> projects and orgs are already moving to 8. These people wonder what our
>> plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
>> >
>> > I’m sure I can dig up a user running Hadoop 0.13 because it ran
>> on JDK5. That doesn’t mean the open source project should stall because
>> certain orgs don’t/can't upgrade.
>> >
>> >>>
>> >>> Drop the 2.6.0 release, branch trunk, and start rolling a
>> >>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>> >>> sustaining work. This gives the rest of the community time to move
>> to JDK8
>> >>> if they haven’t already. For downstream vendors, it gives a roadmap
>> for
>> >>> their customers who will be asking about JDK8 sooner rather than
>> later. By
>> >>> the time 3.0 stabilizes, we’re probably looking at April, which is
>> perfect
>> >>> timing.
>> >>>
>> >>>
>> >> That delays getting stuff out too much; if april slips it becomes a
>> long
>> >> time since an ASF release came out.
>> >
>> > I’m assuming you specifically mean a ‘stable’ release. If, as
>> everyone seems to be saying, that 3.x doesn’t have that much different than
>> 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x
>> took? In other words, if 2.5 is stable and the biggest differences between
>> it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon)
>> that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely
>> unstable? (Thus supporting my conjecture that 2.6 is going to be a
>> problematic release?)
>> >
>> >> Saying "you must run on Java 8 for
>> >> this" will only scare people off and hold back adoption of 3.x,
>> leaving 2.5
>> >> as the last release that ends up being used for a while; the new 1.0.4
>> >
>> > From the outside, trunk looks a lot of 0.21 already. From what
>> I can tell, there is zero motivation to get it out the door and on a
>> roadmap. Primarily because there is little different between trunk and
>> branch-2. This is a very dangerous place to be as those few differences,
>> some measured in years old, rot and wither. :(
>> >
>> >> Here's an alternative
>> >>
>> >> -2.6 on java 6, announce EOL for Java 6 support
>> >> -2.7 on Java 7, state that the lifespan of j7 support will be some
>> bounded
>> >> time period (12-18 mo)
>> >> -trunk to build and test on Java 8 in jenkins alongside java 7. For
>> that to
>> >> be useful someone needs to volunteer to care about build failures. are
>> you
>> >> volunteering, Allen?
>> >
>> > This seems reasonable, except what release should folks who
>> *require* java 8 use? Nightly trunk+patches builds? How do downstream
>> projects test? Should JDK8 fixes be going into a branch? (I’m making the
>> assumption that fixes for JDK8 are not backward compatible with JDK7.
>> Hopefully they are, but given our usage of private APIs…)
>> >
>> > I’ve been approached by a few people over the past month+ if
>> I’d be interested in or will be RM’ing 3.0. I’m seriously considering it
>> esp given a) it’d be a nice learning experience for me b) my “day job”
>> makes it practical time-wise c) I seem to be the only one concerned enough
>> about quite a bit of stale code to get it out the door.
>> >
>> > FWIW, I’m in the process of moving my test vm to JDK8 to see
>> how bad the damage truly is right now. Based on others, it seems security
>> doesn’t work, which is a pretty big deal. I can certainly start posting
>> trunk builds on people.apache.org if folks are interested.
>> >
>> >> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
>> >> through all catch() statements making them multicatch, and the same for
>> >> string case.
>> >
>> > Yup. There’s little reason *not* to switch trunk to JDK7 now.
>>
>
>
--
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Arun Murthy <ac...@hortonworks.com>.
Sorry, coming to discussion late.
We all agreed that 2.6 would the *last* release supporting JDK6 and
hadoop-2.7 would drop support for JDK6. We could easily do 2.7 right after
2.6 (maybe with few critical bug-fixes) with the defining feature of 2.7
being *JDK7 only*. I've checked with HBase, Pig and other communities and
they are good with this. I'm thinking I'll follow up 2-3 wks after 2.6 goes
out to release 2.7 which drops JDK6.
We can certainly add support for JDK8 as early as 2.7 if there are
volunteers - clearly we won't make it depend on JDK8 features right away;
since it would still need to support JDK7.
To recap: hadoop-2.7 onwards would be minimum JDK7, with potential support
for JDK8. We can revisit our discussion from a few months ago to discuss
when we *drop* support for JDK7; clearly something I'd like to avoid doing
in haste.
thanks,
Arun
On Thu, Sep 18, 2014 at 8:41 AM, Alejandro Abdelnur <tu...@gmail.com>
wrote:
> Am I missing something, or we already agreed that after 2.5 release we
> would move trunk and branch-2 to java 7?
>
> On Wed, Sep 17, 2014 at 3:33 PM, Travis Thompson <
> travis.r.thompson@gmail.com> wrote:
>
>> There's actually an umbrella JIRA to track issues with JDK8
>> (HADOOP-11090), in case anyone missed it.
>>
>> At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for
>> about a month now with some mixed results. It definitely works but
>> there are issues, mostly around virtual memory exploding. The reason
>> we took the jump early is there is a company wide push to move to JDK8
>> ASAP, I suspect this isn't something unique to LinkedIn. To get this
>> to work with security enabled, we've had to apply patches not even in
>> trunk yet because they break JDK6 compatibility.
>>
>> From my perspective, based on what I've seen and people I've talked
>> to, there is a huge push to move to JDK8 ASAP so it's becoming
>> increasingly urgent to at least get support to run on JDK8.
>>
>> On Wed, Sep 17, 2014 at 9:55 AM, Allen Wittenauer <aw...@altiscale.com>
>> wrote:
>> >
>> > On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>> >>
>> >> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down
>> the
>> >> filesystem binding with more tests than ever before.
>> >
>> > FWIW, based upon my survey of JIRA, there are a lot of unit
>> test fixes that are only in trunk.
>> >
>> >> But I am also aware of large organisations that are still on Java 6.
>> >> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can
>> help
>> >> them plan.
>> >
>> > Planning is a big thing. That’s one of the reasons why it’d be
>> prudent to start doing 3.0+JDK8 now as well. Even if April slips, other
>> projects and orgs are already moving to 8. These people wonder what our
>> plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
>> >
>> > I’m sure I can dig up a user running Hadoop 0.13 because it ran
>> on JDK5. That doesn’t mean the open source project should stall because
>> certain orgs don’t/can't upgrade.
>> >
>> >>>
>> >>> Drop the 2.6.0 release, branch trunk, and start rolling a
>> >>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>> >>> sustaining work. This gives the rest of the community time to move
>> to JDK8
>> >>> if they haven’t already. For downstream vendors, it gives a roadmap
>> for
>> >>> their customers who will be asking about JDK8 sooner rather than
>> later. By
>> >>> the time 3.0 stabilizes, we’re probably looking at April, which is
>> perfect
>> >>> timing.
>> >>>
>> >>>
>> >> That delays getting stuff out too much; if april slips it becomes a
>> long
>> >> time since an ASF release came out.
>> >
>> > I’m assuming you specifically mean a ‘stable’ release. If, as
>> everyone seems to be saying, that 3.x doesn’t have that much different than
>> 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x
>> took? In other words, if 2.5 is stable and the biggest differences between
>> it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon)
>> that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely
>> unstable? (Thus supporting my conjecture that 2.6 is going to be a
>> problematic release?)
>> >
>> >> Saying "you must run on Java 8 for
>> >> this" will only scare people off and hold back adoption of 3.x,
>> leaving 2.5
>> >> as the last release that ends up being used for a while; the new 1.0.4
>> >
>> > From the outside, trunk looks a lot of 0.21 already. From what
>> I can tell, there is zero motivation to get it out the door and on a
>> roadmap. Primarily because there is little different between trunk and
>> branch-2. This is a very dangerous place to be as those few differences,
>> some measured in years old, rot and wither. :(
>> >
>> >> Here's an alternative
>> >>
>> >> -2.6 on java 6, announce EOL for Java 6 support
>> >> -2.7 on Java 7, state that the lifespan of j7 support will be some
>> bounded
>> >> time period (12-18 mo)
>> >> -trunk to build and test on Java 8 in jenkins alongside java 7. For
>> that to
>> >> be useful someone needs to volunteer to care about build failures. are
>> you
>> >> volunteering, Allen?
>> >
>> > This seems reasonable, except what release should folks who
>> *require* java 8 use? Nightly trunk+patches builds? How do downstream
>> projects test? Should JDK8 fixes be going into a branch? (I’m making the
>> assumption that fixes for JDK8 are not backward compatible with JDK7.
>> Hopefully they are, but given our usage of private APIs…)
>> >
>> > I’ve been approached by a few people over the past month+ if
>> I’d be interested in or will be RM’ing 3.0. I’m seriously considering it
>> esp given a) it’d be a nice learning experience for me b) my “day job”
>> makes it practical time-wise c) I seem to be the only one concerned enough
>> about quite a bit of stale code to get it out the door.
>> >
>> > FWIW, I’m in the process of moving my test vm to JDK8 to see
>> how bad the damage truly is right now. Based on others, it seems security
>> doesn’t work, which is a pretty big deal. I can certainly start posting
>> trunk builds on people.apache.org if folks are interested.
>> >
>> >> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
>> >> through all catch() statements making them multicatch, and the same for
>> >> string case.
>> >
>> > Yup. There’s little reason *not* to switch trunk to JDK7 now.
>>
>
>
--
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Arun Murthy <ac...@hortonworks.com>.
Sorry, coming to discussion late.
We all agreed that 2.6 would the *last* release supporting JDK6 and
hadoop-2.7 would drop support for JDK6. We could easily do 2.7 right after
2.6 (maybe with few critical bug-fixes) with the defining feature of 2.7
being *JDK7 only*. I've checked with HBase, Pig and other communities and
they are good with this. I'm thinking I'll follow up 2-3 wks after 2.6 goes
out to release 2.7 which drops JDK6.
We can certainly add support for JDK8 as early as 2.7 if there are
volunteers - clearly we won't make it depend on JDK8 features right away;
since it would still need to support JDK7.
To recap: hadoop-2.7 onwards would be minimum JDK7, with potential support
for JDK8. We can revisit our discussion from a few months ago to discuss
when we *drop* support for JDK7; clearly something I'd like to avoid doing
in haste.
thanks,
Arun
On Thu, Sep 18, 2014 at 8:41 AM, Alejandro Abdelnur <tu...@gmail.com>
wrote:
> Am I missing something, or we already agreed that after 2.5 release we
> would move trunk and branch-2 to java 7?
>
> On Wed, Sep 17, 2014 at 3:33 PM, Travis Thompson <
> travis.r.thompson@gmail.com> wrote:
>
>> There's actually an umbrella JIRA to track issues with JDK8
>> (HADOOP-11090), in case anyone missed it.
>>
>> At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for
>> about a month now with some mixed results. It definitely works but
>> there are issues, mostly around virtual memory exploding. The reason
>> we took the jump early is there is a company wide push to move to JDK8
>> ASAP, I suspect this isn't something unique to LinkedIn. To get this
>> to work with security enabled, we've had to apply patches not even in
>> trunk yet because they break JDK6 compatibility.
>>
>> From my perspective, based on what I've seen and people I've talked
>> to, there is a huge push to move to JDK8 ASAP so it's becoming
>> increasingly urgent to at least get support to run on JDK8.
>>
>> On Wed, Sep 17, 2014 at 9:55 AM, Allen Wittenauer <aw...@altiscale.com>
>> wrote:
>> >
>> > On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>> >>
>> >> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down
>> the
>> >> filesystem binding with more tests than ever before.
>> >
>> > FWIW, based upon my survey of JIRA, there are a lot of unit
>> test fixes that are only in trunk.
>> >
>> >> But I am also aware of large organisations that are still on Java 6.
>> >> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can
>> help
>> >> them plan.
>> >
>> > Planning is a big thing. That’s one of the reasons why it’d be
>> prudent to start doing 3.0+JDK8 now as well. Even if April slips, other
>> projects and orgs are already moving to 8. These people wonder what our
>> plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
>> >
>> > I’m sure I can dig up a user running Hadoop 0.13 because it ran
>> on JDK5. That doesn’t mean the open source project should stall because
>> certain orgs don’t/can't upgrade.
>> >
>> >>>
>> >>> Drop the 2.6.0 release, branch trunk, and start rolling a
>> >>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>> >>> sustaining work. This gives the rest of the community time to move
>> to JDK8
>> >>> if they haven’t already. For downstream vendors, it gives a roadmap
>> for
>> >>> their customers who will be asking about JDK8 sooner rather than
>> later. By
>> >>> the time 3.0 stabilizes, we’re probably looking at April, which is
>> perfect
>> >>> timing.
>> >>>
>> >>>
>> >> That delays getting stuff out too much; if april slips it becomes a
>> long
>> >> time since an ASF release came out.
>> >
>> > I’m assuming you specifically mean a ‘stable’ release. If, as
>> everyone seems to be saying, that 3.x doesn’t have that much different than
>> 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x
>> took? In other words, if 2.5 is stable and the biggest differences between
>> it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon)
>> that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely
>> unstable? (Thus supporting my conjecture that 2.6 is going to be a
>> problematic release?)
>> >
>> >> Saying "you must run on Java 8 for
>> >> this" will only scare people off and hold back adoption of 3.x,
>> leaving 2.5
>> >> as the last release that ends up being used for a while; the new 1.0.4
>> >
>> > From the outside, trunk looks a lot of 0.21 already. From what
>> I can tell, there is zero motivation to get it out the door and on a
>> roadmap. Primarily because there is little different between trunk and
>> branch-2. This is a very dangerous place to be as those few differences,
>> some measured in years old, rot and wither. :(
>> >
>> >> Here's an alternative
>> >>
>> >> -2.6 on java 6, announce EOL for Java 6 support
>> >> -2.7 on Java 7, state that the lifespan of j7 support will be some
>> bounded
>> >> time period (12-18 mo)
>> >> -trunk to build and test on Java 8 in jenkins alongside java 7. For
>> that to
>> >> be useful someone needs to volunteer to care about build failures. are
>> you
>> >> volunteering, Allen?
>> >
>> > This seems reasonable, except what release should folks who
>> *require* java 8 use? Nightly trunk+patches builds? How do downstream
>> projects test? Should JDK8 fixes be going into a branch? (I’m making the
>> assumption that fixes for JDK8 are not backward compatible with JDK7.
>> Hopefully they are, but given our usage of private APIs…)
>> >
>> > I’ve been approached by a few people over the past month+ if
>> I’d be interested in or will be RM’ing 3.0. I’m seriously considering it
>> esp given a) it’d be a nice learning experience for me b) my “day job”
>> makes it practical time-wise c) I seem to be the only one concerned enough
>> about quite a bit of stale code to get it out the door.
>> >
>> > FWIW, I’m in the process of moving my test vm to JDK8 to see
>> how bad the damage truly is right now. Based on others, it seems security
>> doesn’t work, which is a pretty big deal. I can certainly start posting
>> trunk builds on people.apache.org if folks are interested.
>> >
>> >> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
>> >> through all catch() statements making them multicatch, and the same for
>> >> string case.
>> >
>> > Yup. There’s little reason *not* to switch trunk to JDK7 now.
>>
>
>
--
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Arun Murthy <ac...@hortonworks.com>.
Sorry, coming to discussion late.
We all agreed that 2.6 would the *last* release supporting JDK6 and
hadoop-2.7 would drop support for JDK6. We could easily do 2.7 right after
2.6 (maybe with few critical bug-fixes) with the defining feature of 2.7
being *JDK7 only*. I've checked with HBase, Pig and other communities and
they are good with this. I'm thinking I'll follow up 2-3 wks after 2.6 goes
out to release 2.7 which drops JDK6.
We can certainly add support for JDK8 as early as 2.7 if there are
volunteers - clearly we won't make it depend on JDK8 features right away;
since it would still need to support JDK7.
To recap: hadoop-2.7 onwards would be minimum JDK7, with potential support
for JDK8. We can revisit our discussion from a few months ago to discuss
when we *drop* support for JDK7; clearly something I'd like to avoid doing
in haste.
thanks,
Arun
On Thu, Sep 18, 2014 at 8:41 AM, Alejandro Abdelnur <tu...@gmail.com>
wrote:
> Am I missing something, or we already agreed that after 2.5 release we
> would move trunk and branch-2 to java 7?
>
> On Wed, Sep 17, 2014 at 3:33 PM, Travis Thompson <
> travis.r.thompson@gmail.com> wrote:
>
>> There's actually an umbrella JIRA to track issues with JDK8
>> (HADOOP-11090), in case anyone missed it.
>>
>> At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for
>> about a month now with some mixed results. It definitely works but
>> there are issues, mostly around virtual memory exploding. The reason
>> we took the jump early is there is a company wide push to move to JDK8
>> ASAP, I suspect this isn't something unique to LinkedIn. To get this
>> to work with security enabled, we've had to apply patches not even in
>> trunk yet because they break JDK6 compatibility.
>>
>> From my perspective, based on what I've seen and people I've talked
>> to, there is a huge push to move to JDK8 ASAP so it's becoming
>> increasingly urgent to at least get support to run on JDK8.
>>
>> On Wed, Sep 17, 2014 at 9:55 AM, Allen Wittenauer <aw...@altiscale.com>
>> wrote:
>> >
>> > On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>> >>
>> >> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down
>> the
>> >> filesystem binding with more tests than ever before.
>> >
>> > FWIW, based upon my survey of JIRA, there are a lot of unit
>> test fixes that are only in trunk.
>> >
>> >> But I am also aware of large organisations that are still on Java 6.
>> >> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can
>> help
>> >> them plan.
>> >
>> > Planning is a big thing. That’s one of the reasons why it’d be
>> prudent to start doing 3.0+JDK8 now as well. Even if April slips, other
>> projects and orgs are already moving to 8. These people wonder what our
>> plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
>> >
>> > I’m sure I can dig up a user running Hadoop 0.13 because it ran
>> on JDK5. That doesn’t mean the open source project should stall because
>> certain orgs don’t/can't upgrade.
>> >
>> >>>
>> >>> Drop the 2.6.0 release, branch trunk, and start rolling a
>> >>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>> >>> sustaining work. This gives the rest of the community time to move
>> to JDK8
>> >>> if they haven’t already. For downstream vendors, it gives a roadmap
>> for
>> >>> their customers who will be asking about JDK8 sooner rather than
>> later. By
>> >>> the time 3.0 stabilizes, we’re probably looking at April, which is
>> perfect
>> >>> timing.
>> >>>
>> >>>
>> >> That delays getting stuff out too much; if april slips it becomes a
>> long
>> >> time since an ASF release came out.
>> >
>> > I’m assuming you specifically mean a ‘stable’ release. If, as
>> everyone seems to be saying, that 3.x doesn’t have that much different than
>> 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x
>> took? In other words, if 2.5 is stable and the biggest differences between
>> it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon)
>> that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely
>> unstable? (Thus supporting my conjecture that 2.6 is going to be a
>> problematic release?)
>> >
>> >> Saying "you must run on Java 8 for
>> >> this" will only scare people off and hold back adoption of 3.x,
>> leaving 2.5
>> >> as the last release that ends up being used for a while; the new 1.0.4
>> >
>> > From the outside, trunk looks a lot of 0.21 already. From what
>> I can tell, there is zero motivation to get it out the door and on a
>> roadmap. Primarily because there is little different between trunk and
>> branch-2. This is a very dangerous place to be as those few differences,
>> some measured in years old, rot and wither. :(
>> >
>> >> Here's an alternative
>> >>
>> >> -2.6 on java 6, announce EOL for Java 6 support
>> >> -2.7 on Java 7, state that the lifespan of j7 support will be some
>> bounded
>> >> time period (12-18 mo)
>> >> -trunk to build and test on Java 8 in jenkins alongside java 7. For
>> that to
>> >> be useful someone needs to volunteer to care about build failures. are
>> you
>> >> volunteering, Allen?
>> >
>> > This seems reasonable, except what release should folks who
>> *require* java 8 use? Nightly trunk+patches builds? How do downstream
>> projects test? Should JDK8 fixes be going into a branch? (I’m making the
>> assumption that fixes for JDK8 are not backward compatible with JDK7.
>> Hopefully they are, but given our usage of private APIs…)
>> >
>> > I’ve been approached by a few people over the past month+ if
>> I’d be interested in or will be RM’ing 3.0. I’m seriously considering it
>> esp given a) it’d be a nice learning experience for me b) my “day job”
>> makes it practical time-wise c) I seem to be the only one concerned enough
>> about quite a bit of stale code to get it out the door.
>> >
>> > FWIW, I’m in the process of moving my test vm to JDK8 to see
>> how bad the damage truly is right now. Based on others, it seems security
>> doesn’t work, which is a pretty big deal. I can certainly start posting
>> trunk builds on people.apache.org if folks are interested.
>> >
>> >> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
>> >> through all catch() statements making them multicatch, and the same for
>> >> string case.
>> >
>> > Yup. There’s little reason *not* to switch trunk to JDK7 now.
>>
>
>
--
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Travis Thompson <tr...@gmail.com>.
There's actually an umbrella JIRA to track issues with JDK8
(HADOOP-11090), in case anyone missed it.
At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for
about a month now with some mixed results. It definitely works but
there are issues, mostly around virtual memory exploding. The reason
we took the jump early is there is a company wide push to move to JDK8
ASAP, I suspect this isn't something unique to LinkedIn. To get this
to work with security enabled, we've had to apply patches not even in
trunk yet because they break JDK6 compatibility.
>From my perspective, based on what I've seen and people I've talked
to, there is a huge push to move to JDK8 ASAP so it's becoming
increasingly urgent to at least get support to run on JDK8.
On Wed, Sep 17, 2014 at 9:55 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
>> filesystem binding with more tests than ever before.
>
> FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are only in trunk.
>
>> But I am also aware of large organisations that are still on Java 6.
>> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
>> them plan.
>
> Planning is a big thing. That’s one of the reasons why it’d be prudent to start doing 3.0+JDK8 now as well. Even if April slips, other projects and orgs are already moving to 8. These people wonder what our plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
>
> I’m sure I can dig up a user running Hadoop 0.13 because it ran on JDK5. That doesn’t mean the open source project should stall because certain orgs don’t/can't upgrade.
>
>>>
>>> Drop the 2.6.0 release, branch trunk, and start rolling a
>>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>>> sustaining work. This gives the rest of the community time to move to JDK8
>>> if they haven’t already. For downstream vendors, it gives a roadmap for
>>> their customers who will be asking about JDK8 sooner rather than later. By
>>> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
>>> timing.
>>>
>>>
>> That delays getting stuff out too much; if april slips it becomes a long
>> time since an ASF release came out.
>
> I’m assuming you specifically mean a ‘stable’ release. If, as everyone seems to be saying, that 3.x doesn’t have that much different than 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x took? In other words, if 2.5 is stable and the biggest differences between it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon) that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely unstable? (Thus supporting my conjecture that 2.6 is going to be a problematic release?)
>
>> Saying "you must run on Java 8 for
>> this" will only scare people off and hold back adoption of 3.x, leaving 2.5
>> as the last release that ends up being used for a while; the new 1.0.4
>
> From the outside, trunk looks a lot of 0.21 already. From what I can tell, there is zero motivation to get it out the door and on a roadmap. Primarily because there is little different between trunk and branch-2. This is a very dangerous place to be as those few differences, some measured in years old, rot and wither. :(
>
>> Here's an alternative
>>
>> -2.6 on java 6, announce EOL for Java 6 support
>> -2.7 on Java 7, state that the lifespan of j7 support will be some bounded
>> time period (12-18 mo)
>> -trunk to build and test on Java 8 in jenkins alongside java 7. For that to
>> be useful someone needs to volunteer to care about build failures. are you
>> volunteering, Allen?
>
> This seems reasonable, except what release should folks who *require* java 8 use? Nightly trunk+patches builds? How do downstream projects test? Should JDK8 fixes be going into a branch? (I’m making the assumption that fixes for JDK8 are not backward compatible with JDK7. Hopefully they are, but given our usage of private APIs…)
>
> I’ve been approached by a few people over the past month+ if I’d be interested in or will be RM’ing 3.0. I’m seriously considering it esp given a) it’d be a nice learning experience for me b) my “day job” makes it practical time-wise c) I seem to be the only one concerned enough about quite a bit of stale code to get it out the door.
>
> FWIW, I’m in the process of moving my test vm to JDK8 to see how bad the damage truly is right now. Based on others, it seems security doesn’t work, which is a pretty big deal. I can certainly start posting trunk builds on people.apache.org if folks are interested.
>
>> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
>> through all catch() statements making them multicatch, and the same for
>> string case.
>
> Yup. There’s little reason *not* to switch trunk to JDK7 now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Travis Thompson <tr...@gmail.com>.
There's actually an umbrella JIRA to track issues with JDK8
(HADOOP-11090), in case anyone missed it.
At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for
about a month now with some mixed results. It definitely works but
there are issues, mostly around virtual memory exploding. The reason
we took the jump early is there is a company wide push to move to JDK8
ASAP, I suspect this isn't something unique to LinkedIn. To get this
to work with security enabled, we've had to apply patches not even in
trunk yet because they break JDK6 compatibility.
>From my perspective, based on what I've seen and people I've talked
to, there is a huge push to move to JDK8 ASAP so it's becoming
increasingly urgent to at least get support to run on JDK8.
On Wed, Sep 17, 2014 at 9:55 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
>> filesystem binding with more tests than ever before.
>
> FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are only in trunk.
>
>> But I am also aware of large organisations that are still on Java 6.
>> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
>> them plan.
>
> Planning is a big thing. That’s one of the reasons why it’d be prudent to start doing 3.0+JDK8 now as well. Even if April slips, other projects and orgs are already moving to 8. These people wonder what our plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
>
> I’m sure I can dig up a user running Hadoop 0.13 because it ran on JDK5. That doesn’t mean the open source project should stall because certain orgs don’t/can't upgrade.
>
>>>
>>> Drop the 2.6.0 release, branch trunk, and start rolling a
>>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>>> sustaining work. This gives the rest of the community time to move to JDK8
>>> if they haven’t already. For downstream vendors, it gives a roadmap for
>>> their customers who will be asking about JDK8 sooner rather than later. By
>>> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
>>> timing.
>>>
>>>
>> That delays getting stuff out too much; if april slips it becomes a long
>> time since an ASF release came out.
>
> I’m assuming you specifically mean a ‘stable’ release. If, as everyone seems to be saying, that 3.x doesn’t have that much different than 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x took? In other words, if 2.5 is stable and the biggest differences between it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon) that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely unstable? (Thus supporting my conjecture that 2.6 is going to be a problematic release?)
>
>> Saying "you must run on Java 8 for
>> this" will only scare people off and hold back adoption of 3.x, leaving 2.5
>> as the last release that ends up being used for a while; the new 1.0.4
>
> From the outside, trunk looks a lot of 0.21 already. From what I can tell, there is zero motivation to get it out the door and on a roadmap. Primarily because there is little different between trunk and branch-2. This is a very dangerous place to be as those few differences, some measured in years old, rot and wither. :(
>
>> Here's an alternative
>>
>> -2.6 on java 6, announce EOL for Java 6 support
>> -2.7 on Java 7, state that the lifespan of j7 support will be some bounded
>> time period (12-18 mo)
>> -trunk to build and test on Java 8 in jenkins alongside java 7. For that to
>> be useful someone needs to volunteer to care about build failures. are you
>> volunteering, Allen?
>
> This seems reasonable, except what release should folks who *require* java 8 use? Nightly trunk+patches builds? How do downstream projects test? Should JDK8 fixes be going into a branch? (I’m making the assumption that fixes for JDK8 are not backward compatible with JDK7. Hopefully they are, but given our usage of private APIs…)
>
> I’ve been approached by a few people over the past month+ if I’d be interested in or will be RM’ing 3.0. I’m seriously considering it esp given a) it’d be a nice learning experience for me b) my “day job” makes it practical time-wise c) I seem to be the only one concerned enough about quite a bit of stale code to get it out the door.
>
> FWIW, I’m in the process of moving my test vm to JDK8 to see how bad the damage truly is right now. Based on others, it seems security doesn’t work, which is a pretty big deal. I can certainly start posting trunk builds on people.apache.org if folks are interested.
>
>> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
>> through all catch() statements making them multicatch, and the same for
>> string case.
>
> Yup. There’s little reason *not* to switch trunk to JDK7 now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Travis Thompson <tr...@gmail.com>.
There's actually an umbrella JIRA to track issues with JDK8
(HADOOP-11090), in case anyone missed it.
At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for
about a month now with some mixed results. It definitely works but
there are issues, mostly around virtual memory exploding. The reason
we took the jump early is there is a company wide push to move to JDK8
ASAP, I suspect this isn't something unique to LinkedIn. To get this
to work with security enabled, we've had to apply patches not even in
trunk yet because they break JDK6 compatibility.
>From my perspective, based on what I've seen and people I've talked
to, there is a huge push to move to JDK8 ASAP so it's becoming
increasingly urgent to at least get support to run on JDK8.
On Wed, Sep 17, 2014 at 9:55 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
>> filesystem binding with more tests than ever before.
>
> FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are only in trunk.
>
>> But I am also aware of large organisations that are still on Java 6.
>> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
>> them plan.
>
> Planning is a big thing. That’s one of the reasons why it’d be prudent to start doing 3.0+JDK8 now as well. Even if April slips, other projects and orgs are already moving to 8. These people wonder what our plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
>
> I’m sure I can dig up a user running Hadoop 0.13 because it ran on JDK5. That doesn’t mean the open source project should stall because certain orgs don’t/can't upgrade.
>
>>>
>>> Drop the 2.6.0 release, branch trunk, and start rolling a
>>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>>> sustaining work. This gives the rest of the community time to move to JDK8
>>> if they haven’t already. For downstream vendors, it gives a roadmap for
>>> their customers who will be asking about JDK8 sooner rather than later. By
>>> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
>>> timing.
>>>
>>>
>> That delays getting stuff out too much; if april slips it becomes a long
>> time since an ASF release came out.
>
> I’m assuming you specifically mean a ‘stable’ release. If, as everyone seems to be saying, that 3.x doesn’t have that much different than 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x took? In other words, if 2.5 is stable and the biggest differences between it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon) that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely unstable? (Thus supporting my conjecture that 2.6 is going to be a problematic release?)
>
>> Saying "you must run on Java 8 for
>> this" will only scare people off and hold back adoption of 3.x, leaving 2.5
>> as the last release that ends up being used for a while; the new 1.0.4
>
> From the outside, trunk looks a lot of 0.21 already. From what I can tell, there is zero motivation to get it out the door and on a roadmap. Primarily because there is little different between trunk and branch-2. This is a very dangerous place to be as those few differences, some measured in years old, rot and wither. :(
>
>> Here's an alternative
>>
>> -2.6 on java 6, announce EOL for Java 6 support
>> -2.7 on Java 7, state that the lifespan of j7 support will be some bounded
>> time period (12-18 mo)
>> -trunk to build and test on Java 8 in jenkins alongside java 7. For that to
>> be useful someone needs to volunteer to care about build failures. are you
>> volunteering, Allen?
>
> This seems reasonable, except what release should folks who *require* java 8 use? Nightly trunk+patches builds? How do downstream projects test? Should JDK8 fixes be going into a branch? (I’m making the assumption that fixes for JDK8 are not backward compatible with JDK7. Hopefully they are, but given our usage of private APIs…)
>
> I’ve been approached by a few people over the past month+ if I’d be interested in or will be RM’ing 3.0. I’m seriously considering it esp given a) it’d be a nice learning experience for me b) my “day job” makes it practical time-wise c) I seem to be the only one concerned enough about quite a bit of stale code to get it out the door.
>
> FWIW, I’m in the process of moving my test vm to JDK8 to see how bad the damage truly is right now. Based on others, it seems security doesn’t work, which is a pretty big deal. I can certainly start posting trunk builds on people.apache.org if folks are interested.
>
>> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
>> through all catch() statements making them multicatch, and the same for
>> string case.
>
> Yup. There’s little reason *not* to switch trunk to JDK7 now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Travis Thompson <tr...@gmail.com>.
There's actually an umbrella JIRA to track issues with JDK8
(HADOOP-11090), in case anyone missed it.
At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for
about a month now with some mixed results. It definitely works but
there are issues, mostly around virtual memory exploding. The reason
we took the jump early is there is a company wide push to move to JDK8
ASAP, I suspect this isn't something unique to LinkedIn. To get this
to work with security enabled, we've had to apply patches not even in
trunk yet because they break JDK6 compatibility.
>From my perspective, based on what I've seen and people I've talked
to, there is a huge push to move to JDK8 ASAP so it's becoming
increasingly urgent to at least get support to run on JDK8.
On Wed, Sep 17, 2014 at 9:55 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
>> filesystem binding with more tests than ever before.
>
> FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are only in trunk.
>
>> But I am also aware of large organisations that are still on Java 6.
>> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
>> them plan.
>
> Planning is a big thing. That’s one of the reasons why it’d be prudent to start doing 3.0+JDK8 now as well. Even if April slips, other projects and orgs are already moving to 8. These people wonder what our plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
>
> I’m sure I can dig up a user running Hadoop 0.13 because it ran on JDK5. That doesn’t mean the open source project should stall because certain orgs don’t/can't upgrade.
>
>>>
>>> Drop the 2.6.0 release, branch trunk, and start rolling a
>>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>>> sustaining work. This gives the rest of the community time to move to JDK8
>>> if they haven’t already. For downstream vendors, it gives a roadmap for
>>> their customers who will be asking about JDK8 sooner rather than later. By
>>> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
>>> timing.
>>>
>>>
>> That delays getting stuff out too much; if april slips it becomes a long
>> time since an ASF release came out.
>
> I’m assuming you specifically mean a ‘stable’ release. If, as everyone seems to be saying, that 3.x doesn’t have that much different than 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x took? In other words, if 2.5 is stable and the biggest differences between it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon) that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely unstable? (Thus supporting my conjecture that 2.6 is going to be a problematic release?)
>
>> Saying "you must run on Java 8 for
>> this" will only scare people off and hold back adoption of 3.x, leaving 2.5
>> as the last release that ends up being used for a while; the new 1.0.4
>
> From the outside, trunk looks a lot of 0.21 already. From what I can tell, there is zero motivation to get it out the door and on a roadmap. Primarily because there is little different between trunk and branch-2. This is a very dangerous place to be as those few differences, some measured in years old, rot and wither. :(
>
>> Here's an alternative
>>
>> -2.6 on java 6, announce EOL for Java 6 support
>> -2.7 on Java 7, state that the lifespan of j7 support will be some bounded
>> time period (12-18 mo)
>> -trunk to build and test on Java 8 in jenkins alongside java 7. For that to
>> be useful someone needs to volunteer to care about build failures. are you
>> volunteering, Allen?
>
> This seems reasonable, except what release should folks who *require* java 8 use? Nightly trunk+patches builds? How do downstream projects test? Should JDK8 fixes be going into a branch? (I’m making the assumption that fixes for JDK8 are not backward compatible with JDK7. Hopefully they are, but given our usage of private APIs…)
>
> I’ve been approached by a few people over the past month+ if I’d be interested in or will be RM’ing 3.0. I’m seriously considering it esp given a) it’d be a nice learning experience for me b) my “day job” makes it practical time-wise c) I seem to be the only one concerned enough about quite a bit of stale code to get it out the door.
>
> FWIW, I’m in the process of moving my test vm to JDK8 to see how bad the damage truly is right now. Based on others, it seems security doesn’t work, which is a pretty big deal. I can certainly start posting trunk builds on people.apache.org if folks are interested.
>
>> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
>> through all catch() statements making them multicatch, and the same for
>> string case.
>
> Yup. There’s little reason *not* to switch trunk to JDK7 now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
> filesystem binding with more tests than ever before.
FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are only in trunk.
> But I am also aware of large organisations that are still on Java 6.
> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
> them plan.
Planning is a big thing. That’s one of the reasons why it’d be prudent to start doing 3.0+JDK8 now as well. Even if April slips, other projects and orgs are already moving to 8. These people wonder what our plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
I’m sure I can dig up a user running Hadoop 0.13 because it ran on JDK5. That doesn’t mean the open source project should stall because certain orgs don’t/can't upgrade.
>>
>> Drop the 2.6.0 release, branch trunk, and start rolling a
>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>> sustaining work. This gives the rest of the community time to move to JDK8
>> if they haven’t already. For downstream vendors, it gives a roadmap for
>> their customers who will be asking about JDK8 sooner rather than later. By
>> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
>> timing.
>>
>>
> That delays getting stuff out too much; if april slips it becomes a long
> time since an ASF release came out.
I’m assuming you specifically mean a ‘stable’ release. If, as everyone seems to be saying, that 3.x doesn’t have that much different than 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x took? In other words, if 2.5 is stable and the biggest differences between it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon) that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely unstable? (Thus supporting my conjecture that 2.6 is going to be a problematic release?)
> Saying "you must run on Java 8 for
> this" will only scare people off and hold back adoption of 3.x, leaving 2.5
> as the last release that ends up being used for a while; the new 1.0.4
From the outside, trunk looks a lot of 0.21 already. From what I can tell, there is zero motivation to get it out the door and on a roadmap. Primarily because there is little different between trunk and branch-2. This is a very dangerous place to be as those few differences, some measured in years old, rot and wither. :(
> Here's an alternative
>
> -2.6 on java 6, announce EOL for Java 6 support
> -2.7 on Java 7, state that the lifespan of j7 support will be some bounded
> time period (12-18 mo)
> -trunk to build and test on Java 8 in jenkins alongside java 7. For that to
> be useful someone needs to volunteer to care about build failures. are you
> volunteering, Allen?
This seems reasonable, except what release should folks who *require* java 8 use? Nightly trunk+patches builds? How do downstream projects test? Should JDK8 fixes be going into a branch? (I’m making the assumption that fixes for JDK8 are not backward compatible with JDK7. Hopefully they are, but given our usage of private APIs…)
I’ve been approached by a few people over the past month+ if I’d be interested in or will be RM’ing 3.0. I’m seriously considering it esp given a) it’d be a nice learning experience for me b) my “day job” makes it practical time-wise c) I seem to be the only one concerned enough about quite a bit of stale code to get it out the door.
FWIW, I’m in the process of moving my test vm to JDK8 to see how bad the damage truly is right now. Based on others, it seems security doesn’t work, which is a pretty big deal. I can certainly start posting trunk builds on people.apache.org if folks are interested.
> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
> through all catch() statements making them multicatch, and the same for
> string case.
Yup. There’s little reason *not* to switch trunk to JDK7 now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
> filesystem binding with more tests than ever before.
FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are only in trunk.
> But I am also aware of large organisations that are still on Java 6.
> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
> them plan.
Planning is a big thing. That’s one of the reasons why it’d be prudent to start doing 3.0+JDK8 now as well. Even if April slips, other projects and orgs are already moving to 8. These people wonder what our plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
I’m sure I can dig up a user running Hadoop 0.13 because it ran on JDK5. That doesn’t mean the open source project should stall because certain orgs don’t/can't upgrade.
>>
>> Drop the 2.6.0 release, branch trunk, and start rolling a
>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>> sustaining work. This gives the rest of the community time to move to JDK8
>> if they haven’t already. For downstream vendors, it gives a roadmap for
>> their customers who will be asking about JDK8 sooner rather than later. By
>> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
>> timing.
>>
>>
> That delays getting stuff out too much; if april slips it becomes a long
> time since an ASF release came out.
I’m assuming you specifically mean a ‘stable’ release. If, as everyone seems to be saying, that 3.x doesn’t have that much different than 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x took? In other words, if 2.5 is stable and the biggest differences between it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon) that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely unstable? (Thus supporting my conjecture that 2.6 is going to be a problematic release?)
> Saying "you must run on Java 8 for
> this" will only scare people off and hold back adoption of 3.x, leaving 2.5
> as the last release that ends up being used for a while; the new 1.0.4
From the outside, trunk looks a lot of 0.21 already. From what I can tell, there is zero motivation to get it out the door and on a roadmap. Primarily because there is little different between trunk and branch-2. This is a very dangerous place to be as those few differences, some measured in years old, rot and wither. :(
> Here's an alternative
>
> -2.6 on java 6, announce EOL for Java 6 support
> -2.7 on Java 7, state that the lifespan of j7 support will be some bounded
> time period (12-18 mo)
> -trunk to build and test on Java 8 in jenkins alongside java 7. For that to
> be useful someone needs to volunteer to care about build failures. are you
> volunteering, Allen?
This seems reasonable, except what release should folks who *require* java 8 use? Nightly trunk+patches builds? How do downstream projects test? Should JDK8 fixes be going into a branch? (I’m making the assumption that fixes for JDK8 are not backward compatible with JDK7. Hopefully they are, but given our usage of private APIs…)
I’ve been approached by a few people over the past month+ if I’d be interested in or will be RM’ing 3.0. I’m seriously considering it esp given a) it’d be a nice learning experience for me b) my “day job” makes it practical time-wise c) I seem to be the only one concerned enough about quite a bit of stale code to get it out the door.
FWIW, I’m in the process of moving my test vm to JDK8 to see how bad the damage truly is right now. Based on others, it seems security doesn’t work, which is a pretty big deal. I can certainly start posting trunk builds on people.apache.org if folks are interested.
> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
> through all catch() statements making them multicatch, and the same for
> string case.
Yup. There’s little reason *not* to switch trunk to JDK7 now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
> filesystem binding with more tests than ever before.
FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are only in trunk.
> But I am also aware of large organisations that are still on Java 6.
> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
> them plan.
Planning is a big thing. That’s one of the reasons why it’d be prudent to start doing 3.0+JDK8 now as well. Even if April slips, other projects and orgs are already moving to 8. These people wonder what our plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
I’m sure I can dig up a user running Hadoop 0.13 because it ran on JDK5. That doesn’t mean the open source project should stall because certain orgs don’t/can't upgrade.
>>
>> Drop the 2.6.0 release, branch trunk, and start rolling a
>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>> sustaining work. This gives the rest of the community time to move to JDK8
>> if they haven’t already. For downstream vendors, it gives a roadmap for
>> their customers who will be asking about JDK8 sooner rather than later. By
>> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
>> timing.
>>
>>
> That delays getting stuff out too much; if april slips it becomes a long
> time since an ASF release came out.
I’m assuming you specifically mean a ‘stable’ release. If, as everyone seems to be saying, that 3.x doesn’t have that much different than 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x took? In other words, if 2.5 is stable and the biggest differences between it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon) that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely unstable? (Thus supporting my conjecture that 2.6 is going to be a problematic release?)
> Saying "you must run on Java 8 for
> this" will only scare people off and hold back adoption of 3.x, leaving 2.5
> as the last release that ends up being used for a while; the new 1.0.4
From the outside, trunk looks a lot of 0.21 already. From what I can tell, there is zero motivation to get it out the door and on a roadmap. Primarily because there is little different between trunk and branch-2. This is a very dangerous place to be as those few differences, some measured in years old, rot and wither. :(
> Here's an alternative
>
> -2.6 on java 6, announce EOL for Java 6 support
> -2.7 on Java 7, state that the lifespan of j7 support will be some bounded
> time period (12-18 mo)
> -trunk to build and test on Java 8 in jenkins alongside java 7. For that to
> be useful someone needs to volunteer to care about build failures. are you
> volunteering, Allen?
This seems reasonable, except what release should folks who *require* java 8 use? Nightly trunk+patches builds? How do downstream projects test? Should JDK8 fixes be going into a branch? (I’m making the assumption that fixes for JDK8 are not backward compatible with JDK7. Hopefully they are, but given our usage of private APIs…)
I’ve been approached by a few people over the past month+ if I’d be interested in or will be RM’ing 3.0. I’m seriously considering it esp given a) it’d be a nice learning experience for me b) my “day job” makes it practical time-wise c) I seem to be the only one concerned enough about quite a bit of stale code to get it out the door.
FWIW, I’m in the process of moving my test vm to JDK8 to see how bad the damage truly is right now. Based on others, it seems security doesn’t work, which is a pretty big deal. I can certainly start posting trunk builds on people.apache.org if folks are interested.
> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
> through all catch() statements making them multicatch, and the same for
> string case.
Yup. There’s little reason *not* to switch trunk to JDK7 now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Sep 17, 2014, at 2:47 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
> filesystem binding with more tests than ever before.
FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are only in trunk.
> But I am also aware of large organisations that are still on Java 6.
> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
> them plan.
Planning is a big thing. That’s one of the reasons why it’d be prudent to start doing 3.0+JDK8 now as well. Even if April slips, other projects and orgs are already moving to 8. These people wonder what our plans are so that they can run one JVM. Right now our answer is ¯\_(ツ)_/¯ .
I’m sure I can dig up a user running Hadoop 0.13 because it ran on JDK5. That doesn’t mean the open source project should stall because certain orgs don’t/can't upgrade.
>>
>> Drop the 2.6.0 release, branch trunk, and start rolling a
>> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
>> sustaining work. This gives the rest of the community time to move to JDK8
>> if they haven’t already. For downstream vendors, it gives a roadmap for
>> their customers who will be asking about JDK8 sooner rather than later. By
>> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
>> timing.
>>
>>
> That delays getting stuff out too much; if april slips it becomes a long
> time since an ASF release came out.
I’m assuming you specifically mean a ‘stable’ release. If, as everyone seems to be saying, that 3.x doesn’t have that much different than 2.x, doesn’t this mean that 3.x should be stable much quicker than 2.x took? In other words, if 2.5 is stable and the biggest differences between it and trunk is the majority of code (450+ JIRAs as of yesterday afternoon) that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely unstable? (Thus supporting my conjecture that 2.6 is going to be a problematic release?)
> Saying "you must run on Java 8 for
> this" will only scare people off and hold back adoption of 3.x, leaving 2.5
> as the last release that ends up being used for a while; the new 1.0.4
From the outside, trunk looks a lot of 0.21 already. From what I can tell, there is zero motivation to get it out the door and on a roadmap. Primarily because there is little different between trunk and branch-2. This is a very dangerous place to be as those few differences, some measured in years old, rot and wither. :(
> Here's an alternative
>
> -2.6 on java 6, announce EOL for Java 6 support
> -2.7 on Java 7, state that the lifespan of j7 support will be some bounded
> time period (12-18 mo)
> -trunk to build and test on Java 8 in jenkins alongside java 7. For that to
> be useful someone needs to volunteer to care about build failures. are you
> volunteering, Allen?
This seems reasonable, except what release should folks who *require* java 8 use? Nightly trunk+patches builds? How do downstream projects test? Should JDK8 fixes be going into a branch? (I’m making the assumption that fixes for JDK8 are not backward compatible with JDK7. Hopefully they are, but given our usage of private APIs…)
I’ve been approached by a few people over the past month+ if I’d be interested in or will be RM’ing 3.0. I’m seriously considering it esp given a) it’d be a nice learning experience for me b) my “day job” makes it practical time-wise c) I seem to be the only one concerned enough about quite a bit of stale code to get it out the door.
FWIW, I’m in the process of moving my test vm to JDK8 to see how bad the damage truly is right now. Based on others, it seems security doesn’t work, which is a pretty big deal. I can certainly start posting trunk builds on people.apache.org if folks are interested.
> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
> through all catch() statements making them multicatch, and the same for
> string case.
Yup. There’s little reason *not* to switch trunk to JDK7 now.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Steve Loughran <st...@hortonworks.com>.
On 15 September 2014 18:48, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> It’s now September. With the passage of time, I have a lot of
> doubts about this plan and where that trajectory takes us.
>
> * The list of changes that are already in branch-2 scare the crap out of
> any risk adverse person (Hello to my fellow operations people!). Not only
> are the number of changes extremely high, but in addition there are a lot
> of major, blockbuster features in what is supposed to be a minor release.
> Combined with the fact that we’ve had to do some micro releases, it seems
> to hint that branch-2 is getting less stable over time.
>
I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
filesystem binding with more tests than ever before.
There are changes coming in 2.6, but I see most of them as extensions,
rather than big reworks of the internals
>
> * One of the plans talked about was rolling a 2.7 release that drops JDK6
> and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise
> makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So
> we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too
> late for us to talk about JDK7 and we need to start thinking about JDK8.
>
> I think my stance on JDK7 is well known. I'm personally in favour of
moving to it ASAP, allowing Hadoop to improve, and to allow downstream
projects to move up secure in the knowledge that hadoop 2.7 ==> Java 7+
I also think the language improvements of Java 8 look appealing; we can do
stuff that currently you only gain from by moving to groovy, and with the
improved concurrency stuff you can do it way faster with code is easier to
maintain. A fair few downstream projects use groovy as the test language,
Bigtop and Slider being the two I know of. We get the language improvements
& better assertions, at a price. Java 8 would let us improve the Junit
tests in core hadoop as well as the production code.
But I am also aware of large organisations that are still on Java 6.
Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
them plan.
> * trunk is currently sitting at 3 years old. There is a lot of stuff that
> has been hanging around that really needs to get into people hands so that
> we can start stabilizing it for a “real” release.
>
>
> To me this all says one thing:
>
> Drop the 2.6.0 release, branch trunk, and start rolling a
> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
> sustaining work. This gives the rest of the community time to move to JDK8
> if they haven’t already. For downstream vendors, it gives a roadmap for
> their customers who will be asking about JDK8 sooner rather than later. By
> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
> timing.
>
>
That delays getting stuff out too much; if april slips it becomes a long
time since an ASF release came out. Saying "you must run on Java 8 for
this" will only scare people off and hold back adoption of 3.x, leaving 2.5
as the last release that ends up being used for a while; the new 1.0.4
Here's an alternative
-2.6 on java 6, announce EOL for Java 6 support
-2.7 on Java 7, state that the lifespan of j7 support will be some bounded
time period (12-18 mo)
-trunk to build and test on Java 8 in jenkins alongside java 7. For that to
be useful someone needs to volunteer to care about build failures. are you
volunteering, Allen?
-steve
-we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
through all catch() statements making them multicatch, and the same for
string case.
What it does mean is that trunk can
-adopt the Java 7 APIs where needed for improvements, such as in the native
filesystem IO
-prepare for the java 8 migration by fixing things that don't work on java
8 and which are incompatible with a java 7 codebase.
Doing that in trunk does mean those features can't seamlessly migrate to
hadoop 2.6, so people who hope to do that had better stay in java-6
compatibility mode. Those bits of the code where we know java 6 is
crippling it can move up, announcing it clearly.
> One of the issues I’ve heard mention is that 3.0 doesn’t have
> anything “compelling” in it. Well, dropping 2.6 makes the feature list the
> carrot, JDK8 support is obviously the stick.
>
> Thoughts?
>
>
>
>
> On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan <su...@apache.org>
> wrote:
>
> > Thanks for initiating the thread Arun.
> >
> > Can we add YARN-1051 <https://issues.apache.org/jira/browse/YARN-1051>
> to
> > the list? We have most of the patches for the sub-JIRAs under review and
> > have committed a couple.
> >
> > -Subru
> >
> > ---------- Forwarded message ----------
> >
> > From: Arun C Murthy <ac...@hortonworks.com>
> >
> > Date: Tue, Aug 12, 2014 at 1:34 PM
> >
> > Subject: Thinking ahead to hadoop-2.6
> >
> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> > hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> >
> > "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
> >
> >
> >
> >
> >
> > Folks,
> >
> >
> >
> > With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >
> >
> > Currently, here is the Roadmap per the wiki:
> >
> >
> >
> > • HADOOP
> >
> > • Credential provider HADOOP-10607
> >
> > • HDFS
> >
> > • Heterogeneous storage (Phase 2) - Support APIs for using
> > storage tiers by the applications HDFS-5682
> >
> > • Memory as storage tier HDFS-5851
> >
> > • YARN
> >
> > • Dynamic Resource Configuration YARN-291
> >
> > • NodeManager Restart YARN-1336
> >
> > • ResourceManager HA Phase 2 YARN-556
> >
> > • Support for admin-specified labels in YARN YARN-796
> >
> > • Support for automatic, shared cache for YARN application
> > artifacts YARN-1492
> >
> > • Support NodeGroup layer topology on YARN YARN-18
> >
> > • Support for Docker containers in YARN YARN-1964
> >
> > • YARN service registry YARN-913
> >
> >
> >
> > My suspicion is, as is normal, some will make the cut and some won't.
> >
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >
> >
> > More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> >
> >
> > thanks,
> >
> > Arun
> >
> >
> >
> >
> >
> > --
> >
> > CONFIDENTIALITY NOTICE
> >
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Steve Loughran <st...@hortonworks.com>.
On 15 September 2014 18:48, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> It’s now September. With the passage of time, I have a lot of
> doubts about this plan and where that trajectory takes us.
>
> * The list of changes that are already in branch-2 scare the crap out of
> any risk adverse person (Hello to my fellow operations people!). Not only
> are the number of changes extremely high, but in addition there are a lot
> of major, blockbuster features in what is supposed to be a minor release.
> Combined with the fact that we’ve had to do some micro releases, it seems
> to hint that branch-2 is getting less stable over time.
>
I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
filesystem binding with more tests than ever before.
There are changes coming in 2.6, but I see most of them as extensions,
rather than big reworks of the internals
>
> * One of the plans talked about was rolling a 2.7 release that drops JDK6
> and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise
> makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So
> we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too
> late for us to talk about JDK7 and we need to start thinking about JDK8.
>
> I think my stance on JDK7 is well known. I'm personally in favour of
moving to it ASAP, allowing Hadoop to improve, and to allow downstream
projects to move up secure in the knowledge that hadoop 2.7 ==> Java 7+
I also think the language improvements of Java 8 look appealing; we can do
stuff that currently you only gain from by moving to groovy, and with the
improved concurrency stuff you can do it way faster with code is easier to
maintain. A fair few downstream projects use groovy as the test language,
Bigtop and Slider being the two I know of. We get the language improvements
& better assertions, at a price. Java 8 would let us improve the Junit
tests in core hadoop as well as the production code.
But I am also aware of large organisations that are still on Java 6.
Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
them plan.
> * trunk is currently sitting at 3 years old. There is a lot of stuff that
> has been hanging around that really needs to get into people hands so that
> we can start stabilizing it for a “real” release.
>
>
> To me this all says one thing:
>
> Drop the 2.6.0 release, branch trunk, and start rolling a
> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
> sustaining work. This gives the rest of the community time to move to JDK8
> if they haven’t already. For downstream vendors, it gives a roadmap for
> their customers who will be asking about JDK8 sooner rather than later. By
> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
> timing.
>
>
That delays getting stuff out too much; if april slips it becomes a long
time since an ASF release came out. Saying "you must run on Java 8 for
this" will only scare people off and hold back adoption of 3.x, leaving 2.5
as the last release that ends up being used for a while; the new 1.0.4
Here's an alternative
-2.6 on java 6, announce EOL for Java 6 support
-2.7 on Java 7, state that the lifespan of j7 support will be some bounded
time period (12-18 mo)
-trunk to build and test on Java 8 in jenkins alongside java 7. For that to
be useful someone needs to volunteer to care about build failures. are you
volunteering, Allen?
-steve
-we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
through all catch() statements making them multicatch, and the same for
string case.
What it does mean is that trunk can
-adopt the Java 7 APIs where needed for improvements, such as in the native
filesystem IO
-prepare for the java 8 migration by fixing things that don't work on java
8 and which are incompatible with a java 7 codebase.
Doing that in trunk does mean those features can't seamlessly migrate to
hadoop 2.6, so people who hope to do that had better stay in java-6
compatibility mode. Those bits of the code where we know java 6 is
crippling it can move up, announcing it clearly.
> One of the issues I’ve heard mention is that 3.0 doesn’t have
> anything “compelling” in it. Well, dropping 2.6 makes the feature list the
> carrot, JDK8 support is obviously the stick.
>
> Thoughts?
>
>
>
>
> On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan <su...@apache.org>
> wrote:
>
> > Thanks for initiating the thread Arun.
> >
> > Can we add YARN-1051 <https://issues.apache.org/jira/browse/YARN-1051>
> to
> > the list? We have most of the patches for the sub-JIRAs under review and
> > have committed a couple.
> >
> > -Subru
> >
> > ---------- Forwarded message ----------
> >
> > From: Arun C Murthy <ac...@hortonworks.com>
> >
> > Date: Tue, Aug 12, 2014 at 1:34 PM
> >
> > Subject: Thinking ahead to hadoop-2.6
> >
> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> > hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> >
> > "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
> >
> >
> >
> >
> >
> > Folks,
> >
> >
> >
> > With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >
> >
> > Currently, here is the Roadmap per the wiki:
> >
> >
> >
> > • HADOOP
> >
> > • Credential provider HADOOP-10607
> >
> > • HDFS
> >
> > • Heterogeneous storage (Phase 2) - Support APIs for using
> > storage tiers by the applications HDFS-5682
> >
> > • Memory as storage tier HDFS-5851
> >
> > • YARN
> >
> > • Dynamic Resource Configuration YARN-291
> >
> > • NodeManager Restart YARN-1336
> >
> > • ResourceManager HA Phase 2 YARN-556
> >
> > • Support for admin-specified labels in YARN YARN-796
> >
> > • Support for automatic, shared cache for YARN application
> > artifacts YARN-1492
> >
> > • Support NodeGroup layer topology on YARN YARN-18
> >
> > • Support for Docker containers in YARN YARN-1964
> >
> > • YARN service registry YARN-913
> >
> >
> >
> > My suspicion is, as is normal, some will make the cut and some won't.
> >
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >
> >
> > More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> >
> >
> > thanks,
> >
> > Arun
> >
> >
> >
> >
> >
> > --
> >
> > CONFIDENTIALITY NOTICE
> >
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Steve Loughran <st...@hortonworks.com>.
On 15 September 2014 18:48, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> It’s now September. With the passage of time, I have a lot of
> doubts about this plan and where that trajectory takes us.
>
> * The list of changes that are already in branch-2 scare the crap out of
> any risk adverse person (Hello to my fellow operations people!). Not only
> are the number of changes extremely high, but in addition there are a lot
> of major, blockbuster features in what is supposed to be a minor release.
> Combined with the fact that we’ve had to do some micro releases, it seems
> to hint that branch-2 is getting less stable over time.
>
I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
filesystem binding with more tests than ever before.
There are changes coming in 2.6, but I see most of them as extensions,
rather than big reworks of the internals
>
> * One of the plans talked about was rolling a 2.7 release that drops JDK6
> and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise
> makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So
> we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too
> late for us to talk about JDK7 and we need to start thinking about JDK8.
>
> I think my stance on JDK7 is well known. I'm personally in favour of
moving to it ASAP, allowing Hadoop to improve, and to allow downstream
projects to move up secure in the knowledge that hadoop 2.7 ==> Java 7+
I also think the language improvements of Java 8 look appealing; we can do
stuff that currently you only gain from by moving to groovy, and with the
improved concurrency stuff you can do it way faster with code is easier to
maintain. A fair few downstream projects use groovy as the test language,
Bigtop and Slider being the two I know of. We get the language improvements
& better assertions, at a price. Java 8 would let us improve the Junit
tests in core hadoop as well as the production code.
But I am also aware of large organisations that are still on Java 6.
Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
them plan.
> * trunk is currently sitting at 3 years old. There is a lot of stuff that
> has been hanging around that really needs to get into people hands so that
> we can start stabilizing it for a “real” release.
>
>
> To me this all says one thing:
>
> Drop the 2.6.0 release, branch trunk, and start rolling a
> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
> sustaining work. This gives the rest of the community time to move to JDK8
> if they haven’t already. For downstream vendors, it gives a roadmap for
> their customers who will be asking about JDK8 sooner rather than later. By
> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
> timing.
>
>
That delays getting stuff out too much; if april slips it becomes a long
time since an ASF release came out. Saying "you must run on Java 8 for
this" will only scare people off and hold back adoption of 3.x, leaving 2.5
as the last release that ends up being used for a while; the new 1.0.4
Here's an alternative
-2.6 on java 6, announce EOL for Java 6 support
-2.7 on Java 7, state that the lifespan of j7 support will be some bounded
time period (12-18 mo)
-trunk to build and test on Java 8 in jenkins alongside java 7. For that to
be useful someone needs to volunteer to care about build failures. are you
volunteering, Allen?
-steve
-we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
through all catch() statements making them multicatch, and the same for
string case.
What it does mean is that trunk can
-adopt the Java 7 APIs where needed for improvements, such as in the native
filesystem IO
-prepare for the java 8 migration by fixing things that don't work on java
8 and which are incompatible with a java 7 codebase.
Doing that in trunk does mean those features can't seamlessly migrate to
hadoop 2.6, so people who hope to do that had better stay in java-6
compatibility mode. Those bits of the code where we know java 6 is
crippling it can move up, announcing it clearly.
> One of the issues I’ve heard mention is that 3.0 doesn’t have
> anything “compelling” in it. Well, dropping 2.6 makes the feature list the
> carrot, JDK8 support is obviously the stick.
>
> Thoughts?
>
>
>
>
> On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan <su...@apache.org>
> wrote:
>
> > Thanks for initiating the thread Arun.
> >
> > Can we add YARN-1051 <https://issues.apache.org/jira/browse/YARN-1051>
> to
> > the list? We have most of the patches for the sub-JIRAs under review and
> > have committed a couple.
> >
> > -Subru
> >
> > ---------- Forwarded message ----------
> >
> > From: Arun C Murthy <ac...@hortonworks.com>
> >
> > Date: Tue, Aug 12, 2014 at 1:34 PM
> >
> > Subject: Thinking ahead to hadoop-2.6
> >
> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> > hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> >
> > "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
> >
> >
> >
> >
> >
> > Folks,
> >
> >
> >
> > With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >
> >
> > Currently, here is the Roadmap per the wiki:
> >
> >
> >
> > • HADOOP
> >
> > • Credential provider HADOOP-10607
> >
> > • HDFS
> >
> > • Heterogeneous storage (Phase 2) - Support APIs for using
> > storage tiers by the applications HDFS-5682
> >
> > • Memory as storage tier HDFS-5851
> >
> > • YARN
> >
> > • Dynamic Resource Configuration YARN-291
> >
> > • NodeManager Restart YARN-1336
> >
> > • ResourceManager HA Phase 2 YARN-556
> >
> > • Support for admin-specified labels in YARN YARN-796
> >
> > • Support for automatic, shared cache for YARN application
> > artifacts YARN-1492
> >
> > • Support NodeGroup layer topology on YARN YARN-18
> >
> > • Support for Docker containers in YARN YARN-1964
> >
> > • YARN service registry YARN-913
> >
> >
> >
> > My suspicion is, as is normal, some will make the cut and some won't.
> >
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >
> >
> > More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> >
> >
> > thanks,
> >
> > Arun
> >
> >
> >
> >
> >
> > --
> >
> > CONFIDENTIALITY NOTICE
> >
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> It’s now September. With the passage of time, I have a lot of doubts about this plan and where that trajectory takes us.
>
> * The list of changes that are already in branch-2 scare the crap out of any risk adverse person (Hello to my fellow operations people!). Not only are the number of changes extremely high, but in addition there are a lot of major, blockbuster features in what is supposed to be a minor release. Combined with the fact that we’ve had to do some micro releases, it seems to hint that branch-2 is getting less stable over time.
I don't see what is so scary about 2.6, can you be more concrete? It
seems like a pretty normal release to me and most of the new features
are optional.
I also don't see why you think that "branch-2 is getting less stable
over time." Actually, I think that branch-2 has gotten more stable
over time as people have finally gotten around to upgrading from 1.x
or earlier, and contributed their efforts to addressing regressions in
branch-2.
> * One of the plans talked about was rolling a 2.7 release that drops JDK6 and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too late for us to talk about JDK7 and we need to start thinking about JDK8.
>
> * trunk is currently sitting at 3 years old. There is a lot of stuff that has been hanging around that really needs to get into people hands so that we can start stabilizing it for a “real” release.
We have been pretty careful about minimizing trunk's divergence from
branch-2. I can't think of an example of anything in trunk that
"really needs to get into people's hands"-- did I forget something?
>
>
> To me this all says one thing:
>
> Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all sustaining work. This gives the rest of the community time to move to JDK8 if they haven’t already. For downstream vendors, it gives a roadmap for their customers who will be asking about JDK8 sooner rather than later. By the time 3.0 stabilizes, we’re probably looking at April, which is perfect timing.
>
> One of the issues I’ve heard mention is that 3.0 doesn’t have anything “compelling” in it. Well, dropping 2.6 makes the feature list the carrot, JDK8 support is obviously the stick.
>
> Thoughts?
As we've discussed before, supporting JDK8 is very different from
forcing people to use JDK8. branch-2 and Hadoop 2.6 most certainly
should support JDK8, and most certainly NOT force people to use JDK8.
Cloudera has been using JDK7 internally for a long time, and
recommending it to customers too. Some developers are using JDK8 as
well. It works fine (although I'm sure there will be bugs and
workarounds that get reported and fixed as more people migrate). I
don't see this particular issue as a reason to change the schedule.
best,
Colin
>
>
>
>
> On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan <su...@apache.org> wrote:
>
>> Thanks for initiating the thread Arun.
>>
>> Can we add YARN-1051 <https://issues.apache.org/jira/browse/YARN-1051> to
>> the list? We have most of the patches for the sub-JIRAs under review and
>> have committed a couple.
>>
>> -Subru
>>
>> ---------- Forwarded message ----------
>>
>> From: Arun C Murthy <ac...@hortonworks.com>
>>
>> Date: Tue, Aug 12, 2014 at 1:34 PM
>>
>> Subject: Thinking ahead to hadoop-2.6
>>
>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>> hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>>
>> "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
>>
>>
>>
>>
>>
>> Folks,
>>
>>
>>
>> With hadoop-2.5 nearly done, it's time to start thinking ahead to
>> hadoop-2.6.
>>
>>
>>
>> Currently, here is the Roadmap per the wiki:
>>
>>
>>
>> • HADOOP
>>
>> • Credential provider HADOOP-10607
>>
>> • HDFS
>>
>> • Heterogeneous storage (Phase 2) - Support APIs for using
>> storage tiers by the applications HDFS-5682
>>
>> • Memory as storage tier HDFS-5851
>>
>> • YARN
>>
>> • Dynamic Resource Configuration YARN-291
>>
>> • NodeManager Restart YARN-1336
>>
>> • ResourceManager HA Phase 2 YARN-556
>>
>> • Support for admin-specified labels in YARN YARN-796
>>
>> • Support for automatic, shared cache for YARN application
>> artifacts YARN-1492
>>
>> • Support NodeGroup layer topology on YARN YARN-18
>>
>> • Support for Docker containers in YARN YARN-1964
>>
>> • YARN service registry YARN-913
>>
>>
>>
>> My suspicion is, as is normal, some will make the cut and some won't.
>>
>> Please do add/subtract from the list as appropriate. Ideally, it would be
>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence.
>>
>>
>>
>> More importantly, as we discussed previously, we'd like hadoop-2.6 to be
>> the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
>> discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
>> how they feel about this.
>>
>>
>>
>> thanks,
>>
>> Arun
>>
>>
>>
>>
>>
>> --
>>
>> CONFIDENTIALITY NOTICE
>>
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> It’s now September. With the passage of time, I have a lot of doubts about this plan and where that trajectory takes us.
>
> * The list of changes that are already in branch-2 scare the crap out of any risk adverse person (Hello to my fellow operations people!). Not only are the number of changes extremely high, but in addition there are a lot of major, blockbuster features in what is supposed to be a minor release. Combined with the fact that we’ve had to do some micro releases, it seems to hint that branch-2 is getting less stable over time.
I don't see what is so scary about 2.6, can you be more concrete? It
seems like a pretty normal release to me and most of the new features
are optional.
I also don't see why you think that "branch-2 is getting less stable
over time." Actually, I think that branch-2 has gotten more stable
over time as people have finally gotten around to upgrading from 1.x
or earlier, and contributed their efforts to addressing regressions in
branch-2.
> * One of the plans talked about was rolling a 2.7 release that drops JDK6 and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too late for us to talk about JDK7 and we need to start thinking about JDK8.
>
> * trunk is currently sitting at 3 years old. There is a lot of stuff that has been hanging around that really needs to get into people hands so that we can start stabilizing it for a “real” release.
We have been pretty careful about minimizing trunk's divergence from
branch-2. I can't think of an example of anything in trunk that
"really needs to get into people's hands"-- did I forget something?
>
>
> To me this all says one thing:
>
> Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all sustaining work. This gives the rest of the community time to move to JDK8 if they haven’t already. For downstream vendors, it gives a roadmap for their customers who will be asking about JDK8 sooner rather than later. By the time 3.0 stabilizes, we’re probably looking at April, which is perfect timing.
>
> One of the issues I’ve heard mention is that 3.0 doesn’t have anything “compelling” in it. Well, dropping 2.6 makes the feature list the carrot, JDK8 support is obviously the stick.
>
> Thoughts?
As we've discussed before, supporting JDK8 is very different from
forcing people to use JDK8. branch-2 and Hadoop 2.6 most certainly
should support JDK8, and most certainly NOT force people to use JDK8.
Cloudera has been using JDK7 internally for a long time, and
recommending it to customers too. Some developers are using JDK8 as
well. It works fine (although I'm sure there will be bugs and
workarounds that get reported and fixed as more people migrate). I
don't see this particular issue as a reason to change the schedule.
best,
Colin
>
>
>
>
> On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan <su...@apache.org> wrote:
>
>> Thanks for initiating the thread Arun.
>>
>> Can we add YARN-1051 <https://issues.apache.org/jira/browse/YARN-1051> to
>> the list? We have most of the patches for the sub-JIRAs under review and
>> have committed a couple.
>>
>> -Subru
>>
>> ---------- Forwarded message ----------
>>
>> From: Arun C Murthy <ac...@hortonworks.com>
>>
>> Date: Tue, Aug 12, 2014 at 1:34 PM
>>
>> Subject: Thinking ahead to hadoop-2.6
>>
>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>> hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>>
>> "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
>>
>>
>>
>>
>>
>> Folks,
>>
>>
>>
>> With hadoop-2.5 nearly done, it's time to start thinking ahead to
>> hadoop-2.6.
>>
>>
>>
>> Currently, here is the Roadmap per the wiki:
>>
>>
>>
>> • HADOOP
>>
>> • Credential provider HADOOP-10607
>>
>> • HDFS
>>
>> • Heterogeneous storage (Phase 2) - Support APIs for using
>> storage tiers by the applications HDFS-5682
>>
>> • Memory as storage tier HDFS-5851
>>
>> • YARN
>>
>> • Dynamic Resource Configuration YARN-291
>>
>> • NodeManager Restart YARN-1336
>>
>> • ResourceManager HA Phase 2 YARN-556
>>
>> • Support for admin-specified labels in YARN YARN-796
>>
>> • Support for automatic, shared cache for YARN application
>> artifacts YARN-1492
>>
>> • Support NodeGroup layer topology on YARN YARN-18
>>
>> • Support for Docker containers in YARN YARN-1964
>>
>> • YARN service registry YARN-913
>>
>>
>>
>> My suspicion is, as is normal, some will make the cut and some won't.
>>
>> Please do add/subtract from the list as appropriate. Ideally, it would be
>> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence.
>>
>>
>>
>> More importantly, as we discussed previously, we'd like hadoop-2.6 to be
>> the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
>> discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
>> how they feel about this.
>>
>>
>>
>> thanks,
>>
>> Arun
>>
>>
>>
>>
>>
>> --
>>
>> CONFIDENTIALITY NOTICE
>>
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Posted by Steve Loughran <st...@hortonworks.com>.
On 15 September 2014 18:48, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> It’s now September. With the passage of time, I have a lot of
> doubts about this plan and where that trajectory takes us.
>
> * The list of changes that are already in branch-2 scare the crap out of
> any risk adverse person (Hello to my fellow operations people!). Not only
> are the number of changes extremely high, but in addition there are a lot
> of major, blockbuster features in what is supposed to be a minor release.
> Combined with the fact that we’ve had to do some micro releases, it seems
> to hint that branch-2 is getting less stable over time.
>
I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
filesystem binding with more tests than ever before.
There are changes coming in 2.6, but I see most of them as extensions,
rather than big reworks of the internals
>
> * One of the plans talked about was rolling a 2.7 release that drops JDK6
> and makes JDK7 the standard. If 2.7 comes after 2.6 in October, date wise
> makes it somewhere around January 2015. JDK7 EOL’s in April 2015. So
> we’ll have a viable JDK7 release for exactly 3 months. Frankly, it is too
> late for us to talk about JDK7 and we need to start thinking about JDK8.
>
> I think my stance on JDK7 is well known. I'm personally in favour of
moving to it ASAP, allowing Hadoop to improve, and to allow downstream
projects to move up secure in the knowledge that hadoop 2.7 ==> Java 7+
I also think the language improvements of Java 8 look appealing; we can do
stuff that currently you only gain from by moving to groovy, and with the
improved concurrency stuff you can do it way faster with code is easier to
maintain. A fair few downstream projects use groovy as the test language,
Bigtop and Slider being the two I know of. We get the language improvements
& better assertions, at a price. Java 8 would let us improve the Junit
tests in core hadoop as well as the production code.
But I am also aware of large organisations that are still on Java 6.
Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
them plan.
> * trunk is currently sitting at 3 years old. There is a lot of stuff that
> has been hanging around that really needs to get into people hands so that
> we can start stabilizing it for a “real” release.
>
>
> To me this all says one thing:
>
> Drop the 2.6.0 release, branch trunk, and start rolling a
> 3.0.0-alpha with JDK8 as the minimum. 2.5.1 becomes the base for all
> sustaining work. This gives the rest of the community time to move to JDK8
> if they haven’t already. For downstream vendors, it gives a roadmap for
> their customers who will be asking about JDK8 sooner rather than later. By
> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
> timing.
>
>
That delays getting stuff out too much; if april slips it becomes a long
time since an ASF release came out. Saying "you must run on Java 8 for
this" will only scare people off and hold back adoption of 3.x, leaving 2.5
as the last release that ends up being used for a while; the new 1.0.4
Here's an alternative
-2.6 on java 6, announce EOL for Java 6 support
-2.7 on Java 7, state that the lifespan of j7 support will be some bounded
time period (12-18 mo)
-trunk to build and test on Java 8 in jenkins alongside java 7. For that to
be useful someone needs to volunteer to care about build failures. are you
volunteering, Allen?
-steve
-we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
through all catch() statements making them multicatch, and the same for
string case.
What it does mean is that trunk can
-adopt the Java 7 APIs where needed for improvements, such as in the native
filesystem IO
-prepare for the java 8 migration by fixing things that don't work on java
8 and which are incompatible with a java 7 codebase.
Doing that in trunk does mean those features can't seamlessly migrate to
hadoop 2.6, so people who hope to do that had better stay in java-6
compatibility mode. Those bits of the code where we know java 6 is
crippling it can move up, announcing it clearly.
> One of the issues I’ve heard mention is that 3.0 doesn’t have
> anything “compelling” in it. Well, dropping 2.6 makes the feature list the
> carrot, JDK8 support is obviously the stick.
>
> Thoughts?
>
>
>
>
> On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan <su...@apache.org>
> wrote:
>
> > Thanks for initiating the thread Arun.
> >
> > Can we add YARN-1051 <https://issues.apache.org/jira/browse/YARN-1051>
> to
> > the list? We have most of the patches for the sub-JIRAs under review and
> > have committed a couple.
> >
> > -Subru
> >
> > ---------- Forwarded message ----------
> >
> > From: Arun C Murthy <ac...@hortonworks.com>
> >
> > Date: Tue, Aug 12, 2014 at 1:34 PM
> >
> > Subject: Thinking ahead to hadoop-2.6
> >
> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> > hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> >
> > "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
> >
> >
> >
> >
> >
> > Folks,
> >
> >
> >
> > With hadoop-2.5 nearly done, it's time to start thinking ahead to
> > hadoop-2.6.
> >
> >
> >
> > Currently, here is the Roadmap per the wiki:
> >
> >
> >
> > • HADOOP
> >
> > • Credential provider HADOOP-10607
> >
> > • HDFS
> >
> > • Heterogeneous storage (Phase 2) - Support APIs for using
> > storage tiers by the applications HDFS-5682
> >
> > • Memory as storage tier HDFS-5851
> >
> > • YARN
> >
> > • Dynamic Resource Configuration YARN-291
> >
> > • NodeManager Restart YARN-1336
> >
> > • ResourceManager HA Phase 2 YARN-556
> >
> > • Support for admin-specified labels in YARN YARN-796
> >
> > • Support for automatic, shared cache for YARN application
> > artifacts YARN-1492
> >
> > • Support NodeGroup layer topology on YARN YARN-18
> >
> > • Support for Docker containers in YARN YARN-1964
> >
> > • YARN service registry YARN-913
> >
> >
> >
> > My suspicion is, as is normal, some will make the cut and some won't.
> >
> > Please do add/subtract from the list as appropriate. Ideally, it would be
> > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a
> cadence.
> >
> >
> >
> > More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> > the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> > how they feel about this.
> >
> >
> >
> > thanks,
> >
> > Arun
> >
> >
> >
> >
> >
> > --
> >
> > CONFIDENTIALITY NOTICE
> >
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.