Re: Looking to a Hadoop 3 release

[Andrew Wang <an...@cloudera.com>]

Hi all,

Reviving this thread. I've seen renewed interest in a trunk release since
HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
shell script rewrite, and many other improvements, I think it's time to
revisit Hadoop 3.0 release plans.

My overall plan is still the same as in my original email: a series of
regular alpha releases leading up to beta and GA. Alpha releases make it
easier for downstreams to integrate with our code, and making them regular
means features can be included when they are ready.

I know there are some incompatible changes waiting in the wings
(i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
HADOOP-9991 bumping dependency versions) that would be good to get in. If
you have changes like this, please set the target version to 3.0.0 and mark
them "Incompatible". We can use this JIRA query to track:

https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority

There's some release-related stuff that needs to be sorted out (namely, the
new CHANGES.txt and release note generation from Yetus), but I'd
tentatively like to roll the first alpha a month out, so third week of
March.

Best,
Andrew

On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:

> Avoiding the use of JDK8 language features (and, presumably, APIs)
> means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> source version to JDK8.
>
> Also, note that releasing from trunk is a way of achieving #3, it's
> not a way of abandoning it.
>
>
>
> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> > Hi Raymie,
> >
> > Konst proposed just releasing off of trunk rather than cutting a
> branch-2,
> > and there was general agreement there. So, consider #3 abandoned. 1&2 can
> > be achieved at the same time, we just need to avoid using JDK8 language
> > features in trunk so things can be backported.
> >
> > Best,
> > Andrew
> >
> > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
> wrote:
> >
> >> In this (and the related threads), I see the following three
> requirements:
> >>
> >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> >>
> >> 2. "We'll still be releasing 2.x releases for a while, with similar
> >> feature sets as 3.x."
> >>
> >> 3. Avoid the "risk of split-brain behavior" by "minimize backporting
> >> headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> >> Adding a branch-3, branch-3.x would be obnoxious."
> >>
> >> These three cannot be achieved at the same time.  Which do we abandon?
> >>
> >>
> >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia <sa...@gmail.com>
> >> wrote:
> >> >
> >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org> wrote:
> >> >>
> >> >> 2) Simplification of configs - potentially separating client side
> >> configs
> >> >> and those used by daemons. This is another source of perpetual
> confusion
> >> >> for users.
> >> > + 1 on this.
> >> >
> >> > sanjay
> >>
>

Re: Looking to a Hadoop 3 release

[Andrew Wang <an...@cloudera.com>]

Hi Kai,

Sure, I'm open to it. It's a new major release, so we're allowed to make
these kinds of big changes. The idea behind the extended alpha cycle is
that downstreams can give us feedback. This way if we do anything too
radical, we can address it in the next alpha and have downstreams re-test.

Best,
Andrew

On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai <ka...@intel.com> wrote:

> Thanks Andrew for driving this. Wonder if it's a good chance for
> HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note it's
> not an incompatible change, but feel better to be done in the major release.
>
> Regards,
> Kai
>
> -----Original Message-----
> From: Andrew Wang [mailto:andrew.wang@cloudera.com]
> Sent: Friday, February 19, 2016 7:04 AM
> To: hdfs-dev@hadoop.apache.org; Kihwal Lee <ki...@yahoo-inc.com>
> Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: Looking to a Hadoop 3 release
>
> Hi Kihwal,
>
> I think there's still value in continuing the 2.x releases. 3.x comes with
> the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
> be beta or GA for some number of months. In the meanwhile, it'd be good to
> keep putting out regular, stable 2.x releases.
>
> Best,
> Andrew
>
>
> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
> wrote:
>
> > Moving Hadoop 3 forward sounds fine. If EC is one of the main
> > motivations, are we getting rid of branch-2.8?
> >
> > Kihwal
> >
> >       From: Andrew Wang <an...@cloudera.com>
> >  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> > hdfs-dev <hd...@hadoop.apache.org>
> >  Sent: Thursday, February 18, 2016 4:35 PM
> >  Subject: Re: Looking to a Hadoop 3 release
> >
> > Hi all,
> >
> > Reviving this thread. I've seen renewed interest in a trunk release
> > since HDFS erasure coding has not yet made it to branch-2. Along with
> > JDK8, the shell script rewrite, and many other improvements, I think
> > it's time to revisit Hadoop 3.0 release plans.
> >
> > My overall plan is still the same as in my original email: a series of
> > regular alpha releases leading up to beta and GA. Alpha releases make
> > it easier for downstreams to integrate with our code, and making them
> > regular means features can be included when they are ready.
> >
> > I know there are some incompatible changes waiting in the wings (i.e.
> > HDFS-6984 making FileStatus a PB rather than Writable, some of
> > HADOOP-9991 bumping dependency versions) that would be good to get in.
> > If you have changes like this, please set the target version to 3.0.0
> > and mark them "Incompatible". We can use this JIRA query to track:
> >
> >
> > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> > 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> > 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> > op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
> >
> > There's some release-related stuff that needs to be sorted out
> > (namely, the new CHANGES.txt and release note generation from Yetus),
> > but I'd tentatively like to roll the first alpha a month out, so third
> > week of March.
> >
> > Best,
> > Andrew
> >
> > On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com>
> wrote:
> >
> > > Avoiding the use of JDK8 language features (and, presumably, APIs)
> > > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> > > source version to JDK8.
> > >
> > > Also, note that releasing from trunk is a way of achieving #3, it's
> > > not a way of abandoning it.
> > >
> > >
> > >
> > > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang
> > > <an...@cloudera.com>
> > > wrote:
> > > > Hi Raymie,
> > > >
> > > > Konst proposed just releasing off of trunk rather than cutting a
> > > branch-2,
> > > > and there was general agreement there. So, consider #3 abandoned.
> > > > 1&2
> > can
> > > > be achieved at the same time, we just need to avoid using JDK8
> > > > language features in trunk so things can be backported.
> > > >
> > > > Best,
> > > > Andrew
> > > >
> > > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata
> > > > <rs...@altiscale.com>
> > > wrote:
> > > >
> > > >> In this (and the related threads), I see the following three
> > > requirements:
> > > >>
> > > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > > >>
> > > >> 2. "We'll still be releasing 2.x releases for a while, with
> > > >> similar feature sets as 3.x."
> > > >>
> > > >> 3. Avoid the "risk of split-brain behavior" by "minimize
> > > >> backporting headaches. Pulling trunk > branch-2 > branch-2.x is
> already tedious.
> > > >> Adding a branch-3, branch-3.x would be obnoxious."
> > > >>
> > > >> These three cannot be achieved at the same time.  Which do we
> abandon?
> > > >>
> > > >>
> > > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
> > > >> <sa...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> > wrote:
> > > >> >>
> > > >> >> 2) Simplification of configs - potentially separating client
> > > >> >> side
> > > >> configs
> > > >> >> and those used by daemons. This is another source of perpetual
> > > confusion
> > > >> >> for users.
> > > >> > + 1 on this.
> > > >> >
> > > >> > sanjay
> > > >>
> > >
> >
> >
> >
>

Re: Looking to a Hadoop 3 release

[Andrew Wang <an...@cloudera.com>]

Hi Kai,

Sure, I'm open to it. It's a new major release, so we're allowed to make
these kinds of big changes. The idea behind the extended alpha cycle is
that downstreams can give us feedback. This way if we do anything too
radical, we can address it in the next alpha and have downstreams re-test.

Best,
Andrew

On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai <ka...@intel.com> wrote:

> Thanks Andrew for driving this. Wonder if it's a good chance for
> HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note it's
> not an incompatible change, but feel better to be done in the major release.
>
> Regards,
> Kai
>
> -----Original Message-----
> From: Andrew Wang [mailto:andrew.wang@cloudera.com]
> Sent: Friday, February 19, 2016 7:04 AM
> To: hdfs-dev@hadoop.apache.org; Kihwal Lee <ki...@yahoo-inc.com>
> Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: Looking to a Hadoop 3 release
>
> Hi Kihwal,
>
> I think there's still value in continuing the 2.x releases. 3.x comes with
> the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
> be beta or GA for some number of months. In the meanwhile, it'd be good to
> keep putting out regular, stable 2.x releases.
>
> Best,
> Andrew
>
>
> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
> wrote:
>
> > Moving Hadoop 3 forward sounds fine. If EC is one of the main
> > motivations, are we getting rid of branch-2.8?
> >
> > Kihwal
> >
> >       From: Andrew Wang <an...@cloudera.com>
> >  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> > hdfs-dev <hd...@hadoop.apache.org>
> >  Sent: Thursday, February 18, 2016 4:35 PM
> >  Subject: Re: Looking to a Hadoop 3 release
> >
> > Hi all,
> >
> > Reviving this thread. I've seen renewed interest in a trunk release
> > since HDFS erasure coding has not yet made it to branch-2. Along with
> > JDK8, the shell script rewrite, and many other improvements, I think
> > it's time to revisit Hadoop 3.0 release plans.
> >
> > My overall plan is still the same as in my original email: a series of
> > regular alpha releases leading up to beta and GA. Alpha releases make
> > it easier for downstreams to integrate with our code, and making them
> > regular means features can be included when they are ready.
> >
> > I know there are some incompatible changes waiting in the wings (i.e.
> > HDFS-6984 making FileStatus a PB rather than Writable, some of
> > HADOOP-9991 bumping dependency versions) that would be good to get in.
> > If you have changes like this, please set the target version to 3.0.0
> > and mark them "Incompatible". We can use this JIRA query to track:
> >
> >
> > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> > 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> > 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> > op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
> >
> > There's some release-related stuff that needs to be sorted out
> > (namely, the new CHANGES.txt and release note generation from Yetus),
> > but I'd tentatively like to roll the first alpha a month out, so third
> > week of March.
> >
> > Best,
> > Andrew
> >
> > On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com>
> wrote:
> >
> > > Avoiding the use of JDK8 language features (and, presumably, APIs)
> > > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> > > source version to JDK8.
> > >
> > > Also, note that releasing from trunk is a way of achieving #3, it's
> > > not a way of abandoning it.
> > >
> > >
> > >
> > > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang
> > > <an...@cloudera.com>
> > > wrote:
> > > > Hi Raymie,
> > > >
> > > > Konst proposed just releasing off of trunk rather than cutting a
> > > branch-2,
> > > > and there was general agreement there. So, consider #3 abandoned.
> > > > 1&2
> > can
> > > > be achieved at the same time, we just need to avoid using JDK8
> > > > language features in trunk so things can be backported.
> > > >
> > > > Best,
> > > > Andrew
> > > >
> > > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata
> > > > <rs...@altiscale.com>
> > > wrote:
> > > >
> > > >> In this (and the related threads), I see the following three
> > > requirements:
> > > >>
> > > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > > >>
> > > >> 2. "We'll still be releasing 2.x releases for a while, with
> > > >> similar feature sets as 3.x."
> > > >>
> > > >> 3. Avoid the "risk of split-brain behavior" by "minimize
> > > >> backporting headaches. Pulling trunk > branch-2 > branch-2.x is
> already tedious.
> > > >> Adding a branch-3, branch-3.x would be obnoxious."
> > > >>
> > > >> These three cannot be achieved at the same time.  Which do we
> abandon?
> > > >>
> > > >>
> > > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
> > > >> <sa...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> > wrote:
> > > >> >>
> > > >> >> 2) Simplification of configs - potentially separating client
> > > >> >> side
> > > >> configs
> > > >> >> and those used by daemons. This is another source of perpetual
> > > confusion
> > > >> >> for users.
> > > >> > + 1 on this.
> > > >> >
> > > >> > sanjay
> > > >>
> > >
> >
> >
> >
>

Re: Looking to a Hadoop 3 release

[Andrew Wang <an...@cloudera.com>]

Hi Kai,

Sure, I'm open to it. It's a new major release, so we're allowed to make
these kinds of big changes. The idea behind the extended alpha cycle is
that downstreams can give us feedback. This way if we do anything too
radical, we can address it in the next alpha and have downstreams re-test.

Best,
Andrew

On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai <ka...@intel.com> wrote:

> Thanks Andrew for driving this. Wonder if it's a good chance for
> HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note it's
> not an incompatible change, but feel better to be done in the major release.
>
> Regards,
> Kai
>
> -----Original Message-----
> From: Andrew Wang [mailto:andrew.wang@cloudera.com]
> Sent: Friday, February 19, 2016 7:04 AM
> To: hdfs-dev@hadoop.apache.org; Kihwal Lee <ki...@yahoo-inc.com>
> Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: Looking to a Hadoop 3 release
>
> Hi Kihwal,
>
> I think there's still value in continuing the 2.x releases. 3.x comes with
> the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
> be beta or GA for some number of months. In the meanwhile, it'd be good to
> keep putting out regular, stable 2.x releases.
>
> Best,
> Andrew
>
>
> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
> wrote:
>
> > Moving Hadoop 3 forward sounds fine. If EC is one of the main
> > motivations, are we getting rid of branch-2.8?
> >
> > Kihwal
> >
> >       From: Andrew Wang <an...@cloudera.com>
> >  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> > hdfs-dev <hd...@hadoop.apache.org>
> >  Sent: Thursday, February 18, 2016 4:35 PM
> >  Subject: Re: Looking to a Hadoop 3 release
> >
> > Hi all,
> >
> > Reviving this thread. I've seen renewed interest in a trunk release
> > since HDFS erasure coding has not yet made it to branch-2. Along with
> > JDK8, the shell script rewrite, and many other improvements, I think
> > it's time to revisit Hadoop 3.0 release plans.
> >
> > My overall plan is still the same as in my original email: a series of
> > regular alpha releases leading up to beta and GA. Alpha releases make
> > it easier for downstreams to integrate with our code, and making them
> > regular means features can be included when they are ready.
> >
> > I know there are some incompatible changes waiting in the wings (i.e.
> > HDFS-6984 making FileStatus a PB rather than Writable, some of
> > HADOOP-9991 bumping dependency versions) that would be good to get in.
> > If you have changes like this, please set the target version to 3.0.0
> > and mark them "Incompatible". We can use this JIRA query to track:
> >
> >
> > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> > 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> > 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> > op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
> >
> > There's some release-related stuff that needs to be sorted out
> > (namely, the new CHANGES.txt and release note generation from Yetus),
> > but I'd tentatively like to roll the first alpha a month out, so third
> > week of March.
> >
> > Best,
> > Andrew
> >
> > On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com>
> wrote:
> >
> > > Avoiding the use of JDK8 language features (and, presumably, APIs)
> > > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> > > source version to JDK8.
> > >
> > > Also, note that releasing from trunk is a way of achieving #3, it's
> > > not a way of abandoning it.
> > >
> > >
> > >
> > > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang
> > > <an...@cloudera.com>
> > > wrote:
> > > > Hi Raymie,
> > > >
> > > > Konst proposed just releasing off of trunk rather than cutting a
> > > branch-2,
> > > > and there was general agreement there. So, consider #3 abandoned.
> > > > 1&2
> > can
> > > > be achieved at the same time, we just need to avoid using JDK8
> > > > language features in trunk so things can be backported.
> > > >
> > > > Best,
> > > > Andrew
> > > >
> > > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata
> > > > <rs...@altiscale.com>
> > > wrote:
> > > >
> > > >> In this (and the related threads), I see the following three
> > > requirements:
> > > >>
> > > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > > >>
> > > >> 2. "We'll still be releasing 2.x releases for a while, with
> > > >> similar feature sets as 3.x."
> > > >>
> > > >> 3. Avoid the "risk of split-brain behavior" by "minimize
> > > >> backporting headaches. Pulling trunk > branch-2 > branch-2.x is
> already tedious.
> > > >> Adding a branch-3, branch-3.x would be obnoxious."
> > > >>
> > > >> These three cannot be achieved at the same time.  Which do we
> abandon?
> > > >>
> > > >>
> > > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
> > > >> <sa...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> > wrote:
> > > >> >>
> > > >> >> 2) Simplification of configs - potentially separating client
> > > >> >> side
> > > >> configs
> > > >> >> and those used by daemons. This is another source of perpetual
> > > confusion
> > > >> >> for users.
> > > >> > + 1 on this.
> > > >> >
> > > >> > sanjay
> > > >>
> > >
> >
> >
> >
>

Re: Looking to a Hadoop 3 release

[Andrew Wang <an...@cloudera.com>]

Hi Kai,

Sure, I'm open to it. It's a new major release, so we're allowed to make
these kinds of big changes. The idea behind the extended alpha cycle is
that downstreams can give us feedback. This way if we do anything too
radical, we can address it in the next alpha and have downstreams re-test.

Best,
Andrew

On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai <ka...@intel.com> wrote:

> Thanks Andrew for driving this. Wonder if it's a good chance for
> HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note it's
> not an incompatible change, but feel better to be done in the major release.
>
> Regards,
> Kai
>
> -----Original Message-----
> From: Andrew Wang [mailto:andrew.wang@cloudera.com]
> Sent: Friday, February 19, 2016 7:04 AM
> To: hdfs-dev@hadoop.apache.org; Kihwal Lee <ki...@yahoo-inc.com>
> Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: Looking to a Hadoop 3 release
>
> Hi Kihwal,
>
> I think there's still value in continuing the 2.x releases. 3.x comes with
> the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
> be beta or GA for some number of months. In the meanwhile, it'd be good to
> keep putting out regular, stable 2.x releases.
>
> Best,
> Andrew
>
>
> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
> wrote:
>
> > Moving Hadoop 3 forward sounds fine. If EC is one of the main
> > motivations, are we getting rid of branch-2.8?
> >
> > Kihwal
> >
> >       From: Andrew Wang <an...@cloudera.com>
> >  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> > hdfs-dev <hd...@hadoop.apache.org>
> >  Sent: Thursday, February 18, 2016 4:35 PM
> >  Subject: Re: Looking to a Hadoop 3 release
> >
> > Hi all,
> >
> > Reviving this thread. I've seen renewed interest in a trunk release
> > since HDFS erasure coding has not yet made it to branch-2. Along with
> > JDK8, the shell script rewrite, and many other improvements, I think
> > it's time to revisit Hadoop 3.0 release plans.
> >
> > My overall plan is still the same as in my original email: a series of
> > regular alpha releases leading up to beta and GA. Alpha releases make
> > it easier for downstreams to integrate with our code, and making them
> > regular means features can be included when they are ready.
> >
> > I know there are some incompatible changes waiting in the wings (i.e.
> > HDFS-6984 making FileStatus a PB rather than Writable, some of
> > HADOOP-9991 bumping dependency versions) that would be good to get in.
> > If you have changes like this, please set the target version to 3.0.0
> > and mark them "Incompatible". We can use this JIRA query to track:
> >
> >
> > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> > 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> > 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> > op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
> >
> > There's some release-related stuff that needs to be sorted out
> > (namely, the new CHANGES.txt and release note generation from Yetus),
> > but I'd tentatively like to roll the first alpha a month out, so third
> > week of March.
> >
> > Best,
> > Andrew
> >
> > On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com>
> wrote:
> >
> > > Avoiding the use of JDK8 language features (and, presumably, APIs)
> > > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> > > source version to JDK8.
> > >
> > > Also, note that releasing from trunk is a way of achieving #3, it's
> > > not a way of abandoning it.
> > >
> > >
> > >
> > > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang
> > > <an...@cloudera.com>
> > > wrote:
> > > > Hi Raymie,
> > > >
> > > > Konst proposed just releasing off of trunk rather than cutting a
> > > branch-2,
> > > > and there was general agreement there. So, consider #3 abandoned.
> > > > 1&2
> > can
> > > > be achieved at the same time, we just need to avoid using JDK8
> > > > language features in trunk so things can be backported.
> > > >
> > > > Best,
> > > > Andrew
> > > >
> > > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata
> > > > <rs...@altiscale.com>
> > > wrote:
> > > >
> > > >> In this (and the related threads), I see the following three
> > > requirements:
> > > >>
> > > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > > >>
> > > >> 2. "We'll still be releasing 2.x releases for a while, with
> > > >> similar feature sets as 3.x."
> > > >>
> > > >> 3. Avoid the "risk of split-brain behavior" by "minimize
> > > >> backporting headaches. Pulling trunk > branch-2 > branch-2.x is
> already tedious.
> > > >> Adding a branch-3, branch-3.x would be obnoxious."
> > > >>
> > > >> These three cannot be achieved at the same time.  Which do we
> abandon?
> > > >>
> > > >>
> > > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
> > > >> <sa...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> > wrote:
> > > >> >>
> > > >> >> 2) Simplification of configs - potentially separating client
> > > >> >> side
> > > >> configs
> > > >> >> and those used by daemons. This is another source of perpetual
> > > confusion
> > > >> >> for users.
> > > >> > + 1 on this.
> > > >> >
> > > >> > sanjay
> > > >>
> > >
> >
> >
> >
>

RE: Looking to a Hadoop 3 release

["Zheng, Kai" <ka...@intel.com>]

Thanks Andrew for driving this. Wonder if it's a good chance for HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note it's not an incompatible change, but feel better to be done in the major release.

Regards,
Kai

-----Original Message-----
From: Andrew Wang [mailto:andrew.wang@cloudera.com] 
Sent: Friday, February 19, 2016 7:04 AM
To: hdfs-dev@hadoop.apache.org; Kihwal Lee <ki...@yahoo-inc.com>
Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release

Hi Kihwal,

I think there's still value in continuing the 2.x releases. 3.x comes with the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't be beta or GA for some number of months. In the meanwhile, it'd be good to keep putting out regular, stable 2.x releases.

Best,
Andrew


On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:

> Moving Hadoop 3 forward sounds fine. If EC is one of the main 
> motivations, are we getting rid of branch-2.8?
>
> Kihwal
>
>       From: Andrew Wang <an...@cloudera.com>
>  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> hdfs-dev <hd...@hadoop.apache.org>
>  Sent: Thursday, February 18, 2016 4:35 PM
>  Subject: Re: Looking to a Hadoop 3 release
>
> Hi all,
>
> Reviving this thread. I've seen renewed interest in a trunk release 
> since HDFS erasure coding has not yet made it to branch-2. Along with 
> JDK8, the shell script rewrite, and many other improvements, I think 
> it's time to revisit Hadoop 3.0 release plans.
>
> My overall plan is still the same as in my original email: a series of 
> regular alpha releases leading up to beta and GA. Alpha releases make 
> it easier for downstreams to integrate with our code, and making them 
> regular means features can be included when they are ready.
>
> I know there are some incompatible changes waiting in the wings (i.e. 
> HDFS-6984 making FileStatus a PB rather than Writable, some of
> HADOOP-9991 bumping dependency versions) that would be good to get in. 
> If you have changes like this, please set the target version to 3.0.0 
> and mark them "Incompatible". We can use this JIRA query to track:
>
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
>
> There's some release-related stuff that needs to be sorted out 
> (namely, the new CHANGES.txt and release note generation from Yetus), 
> but I'd tentatively like to roll the first alpha a month out, so third 
> week of March.
>
> Best,
> Andrew
>
> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:
>
> > Avoiding the use of JDK8 language features (and, presumably, APIs) 
> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK 
> > source version to JDK8.
> >
> > Also, note that releasing from trunk is a way of achieving #3, it's 
> > not a way of abandoning it.
> >
> >
> >
> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang 
> > <an...@cloudera.com>
> > wrote:
> > > Hi Raymie,
> > >
> > > Konst proposed just releasing off of trunk rather than cutting a
> > branch-2,
> > > and there was general agreement there. So, consider #3 abandoned. 
> > > 1&2
> can
> > > be achieved at the same time, we just need to avoid using JDK8 
> > > language features in trunk so things can be backported.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata 
> > > <rs...@altiscale.com>
> > wrote:
> > >
> > >> In this (and the related threads), I see the following three
> > requirements:
> > >>
> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > >>
> > >> 2. "We'll still be releasing 2.x releases for a while, with 
> > >> similar feature sets as 3.x."
> > >>
> > >> 3. Avoid the "risk of split-brain behavior" by "minimize 
> > >> backporting headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> > >> Adding a branch-3, branch-3.x would be obnoxious."
> > >>
> > >> These three cannot be achieved at the same time.  Which do we abandon?
> > >>
> > >>
> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia 
> > >> <sa...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> > >> >>
> > >> >> 2) Simplification of configs - potentially separating client 
> > >> >> side
> > >> configs
> > >> >> and those used by daemons. This is another source of perpetual
> > confusion
> > >> >> for users.
> > >> > + 1 on this.
> > >> >
> > >> > sanjay
> > >>
> >
>
>
>

RE: Looking to a Hadoop 3 release

["Zheng, Kai" <ka...@intel.com>]

Thanks Andrew for driving this. Wonder if it's a good chance for HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note it's not an incompatible change, but feel better to be done in the major release.

Regards,
Kai

-----Original Message-----
From: Andrew Wang [mailto:andrew.wang@cloudera.com] 
Sent: Friday, February 19, 2016 7:04 AM
To: hdfs-dev@hadoop.apache.org; Kihwal Lee <ki...@yahoo-inc.com>
Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release

Hi Kihwal,

I think there's still value in continuing the 2.x releases. 3.x comes with the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't be beta or GA for some number of months. In the meanwhile, it'd be good to keep putting out regular, stable 2.x releases.

Best,
Andrew


On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:

> Moving Hadoop 3 forward sounds fine. If EC is one of the main 
> motivations, are we getting rid of branch-2.8?
>
> Kihwal
>
>       From: Andrew Wang <an...@cloudera.com>
>  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> hdfs-dev <hd...@hadoop.apache.org>
>  Sent: Thursday, February 18, 2016 4:35 PM
>  Subject: Re: Looking to a Hadoop 3 release
>
> Hi all,
>
> Reviving this thread. I've seen renewed interest in a trunk release 
> since HDFS erasure coding has not yet made it to branch-2. Along with 
> JDK8, the shell script rewrite, and many other improvements, I think 
> it's time to revisit Hadoop 3.0 release plans.
>
> My overall plan is still the same as in my original email: a series of 
> regular alpha releases leading up to beta and GA. Alpha releases make 
> it easier for downstreams to integrate with our code, and making them 
> regular means features can be included when they are ready.
>
> I know there are some incompatible changes waiting in the wings (i.e. 
> HDFS-6984 making FileStatus a PB rather than Writable, some of
> HADOOP-9991 bumping dependency versions) that would be good to get in. 
> If you have changes like this, please set the target version to 3.0.0 
> and mark them "Incompatible". We can use this JIRA query to track:
>
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
>
> There's some release-related stuff that needs to be sorted out 
> (namely, the new CHANGES.txt and release note generation from Yetus), 
> but I'd tentatively like to roll the first alpha a month out, so third 
> week of March.
>
> Best,
> Andrew
>
> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:
>
> > Avoiding the use of JDK8 language features (and, presumably, APIs) 
> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK 
> > source version to JDK8.
> >
> > Also, note that releasing from trunk is a way of achieving #3, it's 
> > not a way of abandoning it.
> >
> >
> >
> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang 
> > <an...@cloudera.com>
> > wrote:
> > > Hi Raymie,
> > >
> > > Konst proposed just releasing off of trunk rather than cutting a
> > branch-2,
> > > and there was general agreement there. So, consider #3 abandoned. 
> > > 1&2
> can
> > > be achieved at the same time, we just need to avoid using JDK8 
> > > language features in trunk so things can be backported.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata 
> > > <rs...@altiscale.com>
> > wrote:
> > >
> > >> In this (and the related threads), I see the following three
> > requirements:
> > >>
> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > >>
> > >> 2. "We'll still be releasing 2.x releases for a while, with 
> > >> similar feature sets as 3.x."
> > >>
> > >> 3. Avoid the "risk of split-brain behavior" by "minimize 
> > >> backporting headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> > >> Adding a branch-3, branch-3.x would be obnoxious."
> > >>
> > >> These three cannot be achieved at the same time.  Which do we abandon?
> > >>
> > >>
> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia 
> > >> <sa...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> > >> >>
> > >> >> 2) Simplification of configs - potentially separating client 
> > >> >> side
> > >> configs
> > >> >> and those used by daemons. This is another source of perpetual
> > confusion
> > >> >> for users.
> > >> > + 1 on this.
> > >> >
> > >> > sanjay
> > >>
> >
>
>
>

RE: Looking to a Hadoop 3 release

["Zheng, Kai" <ka...@intel.com>]

Thanks Andrew for driving this. Wonder if it's a good chance for HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note it's not an incompatible change, but feel better to be done in the major release.

Regards,
Kai

-----Original Message-----
From: Andrew Wang [mailto:andrew.wang@cloudera.com] 
Sent: Friday, February 19, 2016 7:04 AM
To: hdfs-dev@hadoop.apache.org; Kihwal Lee <ki...@yahoo-inc.com>
Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release

Hi Kihwal,

I think there's still value in continuing the 2.x releases. 3.x comes with the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't be beta or GA for some number of months. In the meanwhile, it'd be good to keep putting out regular, stable 2.x releases.

Best,
Andrew


On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:

> Moving Hadoop 3 forward sounds fine. If EC is one of the main 
> motivations, are we getting rid of branch-2.8?
>
> Kihwal
>
>       From: Andrew Wang <an...@cloudera.com>
>  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> hdfs-dev <hd...@hadoop.apache.org>
>  Sent: Thursday, February 18, 2016 4:35 PM
>  Subject: Re: Looking to a Hadoop 3 release
>
> Hi all,
>
> Reviving this thread. I've seen renewed interest in a trunk release 
> since HDFS erasure coding has not yet made it to branch-2. Along with 
> JDK8, the shell script rewrite, and many other improvements, I think 
> it's time to revisit Hadoop 3.0 release plans.
>
> My overall plan is still the same as in my original email: a series of 
> regular alpha releases leading up to beta and GA. Alpha releases make 
> it easier for downstreams to integrate with our code, and making them 
> regular means features can be included when they are ready.
>
> I know there are some incompatible changes waiting in the wings (i.e. 
> HDFS-6984 making FileStatus a PB rather than Writable, some of
> HADOOP-9991 bumping dependency versions) that would be good to get in. 
> If you have changes like this, please set the target version to 3.0.0 
> and mark them "Incompatible". We can use this JIRA query to track:
>
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
>
> There's some release-related stuff that needs to be sorted out 
> (namely, the new CHANGES.txt and release note generation from Yetus), 
> but I'd tentatively like to roll the first alpha a month out, so third 
> week of March.
>
> Best,
> Andrew
>
> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:
>
> > Avoiding the use of JDK8 language features (and, presumably, APIs) 
> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK 
> > source version to JDK8.
> >
> > Also, note that releasing from trunk is a way of achieving #3, it's 
> > not a way of abandoning it.
> >
> >
> >
> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang 
> > <an...@cloudera.com>
> > wrote:
> > > Hi Raymie,
> > >
> > > Konst proposed just releasing off of trunk rather than cutting a
> > branch-2,
> > > and there was general agreement there. So, consider #3 abandoned. 
> > > 1&2
> can
> > > be achieved at the same time, we just need to avoid using JDK8 
> > > language features in trunk so things can be backported.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata 
> > > <rs...@altiscale.com>
> > wrote:
> > >
> > >> In this (and the related threads), I see the following three
> > requirements:
> > >>
> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > >>
> > >> 2. "We'll still be releasing 2.x releases for a while, with 
> > >> similar feature sets as 3.x."
> > >>
> > >> 3. Avoid the "risk of split-brain behavior" by "minimize 
> > >> backporting headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> > >> Adding a branch-3, branch-3.x would be obnoxious."
> > >>
> > >> These three cannot be achieved at the same time.  Which do we abandon?
> > >>
> > >>
> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia 
> > >> <sa...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> > >> >>
> > >> >> 2) Simplification of configs - potentially separating client 
> > >> >> side
> > >> configs
> > >> >> and those used by daemons. This is another source of perpetual
> > confusion
> > >> >> for users.
> > >> > + 1 on this.
> > >> >
> > >> > sanjay
> > >>
> >
>
>
>

Re: Looking to a Hadoop 3 release

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

+1 for the 3.0 release plan and continuing 2.x releases.
I'm thinking we should consider stopping new 2.x minor releases after 
3.x reaches GA.

Thanks,
Akira

On 2/19/16 10:33, Gangumalla, Uma wrote:
> Yes. I think starting 3.0 release with alpha is good idea. So it would get
> some time to reach the beta or GA.
>
> +1 for the plan.
>
> For the compatibility purposes and as current stable versions, we should
> continue 2.x releases anyway.
>
> Thanks Andrew for starting the thread.
>
> Regards,
> Uma
>
> On 2/18/16, 3:04 PM, "Andrew Wang" <an...@cloudera.com> wrote:
>
>> Hi Kihwal,
>>
>> I think there's still value in continuing the 2.x releases. 3.x comes with
>> the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
>> be beta or GA for some number of months. In the meanwhile, it'd be good to
>> keep putting out regular, stable 2.x releases.
>>
>> Best,
>> Andrew
>>
>>
>> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
>> wrote:
>>
>>> Moving Hadoop 3 forward sounds fine. If EC is one of the main
>>> motivations,
>>> are we getting rid of branch-2.8?
>>>
>>> Kihwal
>>>
>>>        From: Andrew Wang <an...@cloudera.com>
>>>   To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>>> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
>>> hdfs-dev <hd...@hadoop.apache.org>
>>>   Sent: Thursday, February 18, 2016 4:35 PM
>>>   Subject: Re: Looking to a Hadoop 3 release
>>>
>>> Hi all,
>>>
>>> Reviving this thread. I've seen renewed interest in a trunk release
>>> since
>>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
>>> the
>>> shell script rewrite, and many other improvements, I think it's time to
>>> revisit Hadoop 3.0 release plans.
>>>
>>> My overall plan is still the same as in my original email: a series of
>>> regular alpha releases leading up to beta and GA. Alpha releases make it
>>> easier for downstreams to integrate with our code, and making them
>>> regular
>>> means features can be included when they are ready.
>>>
>>> I know there are some incompatible changes waiting in the wings
>>> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
>>> HADOOP-9991 bumping dependency versions) that would be good to get in.
>>> If
>>> you have changes like this, please set the target version to 3.0.0 and
>>> mark
>>> them "Incompatible". We can use this JIRA query to track:
>>>
>>>
>>>
>>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HD
>>> FS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%
>>> 223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flag
>>> s%22%3D%22Incompatible%20change%22%20order%20by%20priority
>>>
>>> There's some release-related stuff that needs to be sorted out (namely,
>>> the
>>> new CHANGES.txt and release note generation from Yetus), but I'd
>>> tentatively like to roll the first alpha a month out, so third week of
>>> March.
>>>
>>> Best,
>>> Andrew
>>>
>>> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com>
>>> wrote:
>>>
>>>> Avoiding the use of JDK8 language features (and, presumably, APIs)
>>>> means you've abandoned #1, i.e., you haven't (really) bumped the JDK
>>>> source version to JDK8.
>>>>
>>>> Also, note that releasing from trunk is a way of achieving #3, it's
>>>> not a way of abandoning it.
>>>>
>>>>
>>>>
>>>> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>>> Hi Raymie,
>>>>>
>>>>> Konst proposed just releasing off of trunk rather than cutting a
>>>> branch-2,
>>>>> and there was general agreement there. So, consider #3 abandoned.
>>> 1&2
>>> can
>>>>> be achieved at the same time, we just need to avoid using JDK8
>>> language
>>>>> features in trunk so things can be backported.
>>>>>
>>>>> Best,
>>>>> Andrew
>>>>>
>>>>> On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
>>>> wrote:
>>>>>
>>>>>> In this (and the related threads), I see the following three
>>>> requirements:
>>>>>>
>>>>>> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
>>>>>>
>>>>>> 2. "We'll still be releasing 2.x releases for a while, with similar
>>>>>> feature sets as 3.x."
>>>>>>
>>>>>> 3. Avoid the "risk of split-brain behavior" by "minimize
>>> backporting
>>>>>> headaches. Pulling trunk > branch-2 > branch-2.x is already
>>> tedious.
>>>>>> Adding a branch-3, branch-3.x would be obnoxious."
>>>>>>
>>>>>> These three cannot be achieved at the same time.  Which do we
>>> abandon?
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
>>> <sa...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
>>> wrote:
>>>>>>>>
>>>>>>>> 2) Simplification of configs - potentially separating client
>>> side
>>>>>> configs
>>>>>>>> and those used by daemons. This is another source of perpetual
>>>> confusion
>>>>>>>> for users.
>>>>>>> + 1 on this.
>>>>>>>
>>>>>>> sanjay
>>>>>>
>>>>
>>>
>>>
>>>
>

Re: Looking to a Hadoop 3 release

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

+1 for the 3.0 release plan and continuing 2.x releases.
I'm thinking we should consider stopping new 2.x minor releases after 
3.x reaches GA.

Thanks,
Akira

On 2/19/16 10:33, Gangumalla, Uma wrote:
> Yes. I think starting 3.0 release with alpha is good idea. So it would get
> some time to reach the beta or GA.
>
> +1 for the plan.
>
> For the compatibility purposes and as current stable versions, we should
> continue 2.x releases anyway.
>
> Thanks Andrew for starting the thread.
>
> Regards,
> Uma
>
> On 2/18/16, 3:04 PM, "Andrew Wang" <an...@cloudera.com> wrote:
>
>> Hi Kihwal,
>>
>> I think there's still value in continuing the 2.x releases. 3.x comes with
>> the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
>> be beta or GA for some number of months. In the meanwhile, it'd be good to
>> keep putting out regular, stable 2.x releases.
>>
>> Best,
>> Andrew
>>
>>
>> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
>> wrote:
>>
>>> Moving Hadoop 3 forward sounds fine. If EC is one of the main
>>> motivations,
>>> are we getting rid of branch-2.8?
>>>
>>> Kihwal
>>>
>>>        From: Andrew Wang <an...@cloudera.com>
>>>   To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>>> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
>>> hdfs-dev <hd...@hadoop.apache.org>
>>>   Sent: Thursday, February 18, 2016 4:35 PM
>>>   Subject: Re: Looking to a Hadoop 3 release
>>>
>>> Hi all,
>>>
>>> Reviving this thread. I've seen renewed interest in a trunk release
>>> since
>>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
>>> the
>>> shell script rewrite, and many other improvements, I think it's time to
>>> revisit Hadoop 3.0 release plans.
>>>
>>> My overall plan is still the same as in my original email: a series of
>>> regular alpha releases leading up to beta and GA. Alpha releases make it
>>> easier for downstreams to integrate with our code, and making them
>>> regular
>>> means features can be included when they are ready.
>>>
>>> I know there are some incompatible changes waiting in the wings
>>> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
>>> HADOOP-9991 bumping dependency versions) that would be good to get in.
>>> If
>>> you have changes like this, please set the target version to 3.0.0 and
>>> mark
>>> them "Incompatible". We can use this JIRA query to track:
>>>
>>>
>>>
>>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HD
>>> FS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%
>>> 223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flag
>>> s%22%3D%22Incompatible%20change%22%20order%20by%20priority
>>>
>>> There's some release-related stuff that needs to be sorted out (namely,
>>> the
>>> new CHANGES.txt and release note generation from Yetus), but I'd
>>> tentatively like to roll the first alpha a month out, so third week of
>>> March.
>>>
>>> Best,
>>> Andrew
>>>
>>> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com>
>>> wrote:
>>>
>>>> Avoiding the use of JDK8 language features (and, presumably, APIs)
>>>> means you've abandoned #1, i.e., you haven't (really) bumped the JDK
>>>> source version to JDK8.
>>>>
>>>> Also, note that releasing from trunk is a way of achieving #3, it's
>>>> not a way of abandoning it.
>>>>
>>>>
>>>>
>>>> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>>> Hi Raymie,
>>>>>
>>>>> Konst proposed just releasing off of trunk rather than cutting a
>>>> branch-2,
>>>>> and there was general agreement there. So, consider #3 abandoned.
>>> 1&2
>>> can
>>>>> be achieved at the same time, we just need to avoid using JDK8
>>> language
>>>>> features in trunk so things can be backported.
>>>>>
>>>>> Best,
>>>>> Andrew
>>>>>
>>>>> On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
>>>> wrote:
>>>>>
>>>>>> In this (and the related threads), I see the following three
>>>> requirements:
>>>>>>
>>>>>> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
>>>>>>
>>>>>> 2. "We'll still be releasing 2.x releases for a while, with similar
>>>>>> feature sets as 3.x."
>>>>>>
>>>>>> 3. Avoid the "risk of split-brain behavior" by "minimize
>>> backporting
>>>>>> headaches. Pulling trunk > branch-2 > branch-2.x is already
>>> tedious.
>>>>>> Adding a branch-3, branch-3.x would be obnoxious."
>>>>>>
>>>>>> These three cannot be achieved at the same time.  Which do we
>>> abandon?
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
>>> <sa...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
>>> wrote:
>>>>>>>>
>>>>>>>> 2) Simplification of configs - potentially separating client
>>> side
>>>>>> configs
>>>>>>>> and those used by daemons. This is another source of perpetual
>>>> confusion
>>>>>>>> for users.
>>>>>>> + 1 on this.
>>>>>>>
>>>>>>> sanjay
>>>>>>
>>>>
>>>
>>>
>>>
>

Re: Looking to a Hadoop 3 release

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

+1 for the 3.0 release plan and continuing 2.x releases.
I'm thinking we should consider stopping new 2.x minor releases after 
3.x reaches GA.

Thanks,
Akira

On 2/19/16 10:33, Gangumalla, Uma wrote:
> Yes. I think starting 3.0 release with alpha is good idea. So it would get
> some time to reach the beta or GA.
>
> +1 for the plan.
>
> For the compatibility purposes and as current stable versions, we should
> continue 2.x releases anyway.
>
> Thanks Andrew for starting the thread.
>
> Regards,
> Uma
>
> On 2/18/16, 3:04 PM, "Andrew Wang" <an...@cloudera.com> wrote:
>
>> Hi Kihwal,
>>
>> I think there's still value in continuing the 2.x releases. 3.x comes with
>> the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
>> be beta or GA for some number of months. In the meanwhile, it'd be good to
>> keep putting out regular, stable 2.x releases.
>>
>> Best,
>> Andrew
>>
>>
>> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
>> wrote:
>>
>>> Moving Hadoop 3 forward sounds fine. If EC is one of the main
>>> motivations,
>>> are we getting rid of branch-2.8?
>>>
>>> Kihwal
>>>
>>>        From: Andrew Wang <an...@cloudera.com>
>>>   To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>>> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
>>> hdfs-dev <hd...@hadoop.apache.org>
>>>   Sent: Thursday, February 18, 2016 4:35 PM
>>>   Subject: Re: Looking to a Hadoop 3 release
>>>
>>> Hi all,
>>>
>>> Reviving this thread. I've seen renewed interest in a trunk release
>>> since
>>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
>>> the
>>> shell script rewrite, and many other improvements, I think it's time to
>>> revisit Hadoop 3.0 release plans.
>>>
>>> My overall plan is still the same as in my original email: a series of
>>> regular alpha releases leading up to beta and GA. Alpha releases make it
>>> easier for downstreams to integrate with our code, and making them
>>> regular
>>> means features can be included when they are ready.
>>>
>>> I know there are some incompatible changes waiting in the wings
>>> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
>>> HADOOP-9991 bumping dependency versions) that would be good to get in.
>>> If
>>> you have changes like this, please set the target version to 3.0.0 and
>>> mark
>>> them "Incompatible". We can use this JIRA query to track:
>>>
>>>
>>>
>>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HD
>>> FS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%
>>> 223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flag
>>> s%22%3D%22Incompatible%20change%22%20order%20by%20priority
>>>
>>> There's some release-related stuff that needs to be sorted out (namely,
>>> the
>>> new CHANGES.txt and release note generation from Yetus), but I'd
>>> tentatively like to roll the first alpha a month out, so third week of
>>> March.
>>>
>>> Best,
>>> Andrew
>>>
>>> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com>
>>> wrote:
>>>
>>>> Avoiding the use of JDK8 language features (and, presumably, APIs)
>>>> means you've abandoned #1, i.e., you haven't (really) bumped the JDK
>>>> source version to JDK8.
>>>>
>>>> Also, note that releasing from trunk is a way of achieving #3, it's
>>>> not a way of abandoning it.
>>>>
>>>>
>>>>
>>>> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>>> Hi Raymie,
>>>>>
>>>>> Konst proposed just releasing off of trunk rather than cutting a
>>>> branch-2,
>>>>> and there was general agreement there. So, consider #3 abandoned.
>>> 1&2
>>> can
>>>>> be achieved at the same time, we just need to avoid using JDK8
>>> language
>>>>> features in trunk so things can be backported.
>>>>>
>>>>> Best,
>>>>> Andrew
>>>>>
>>>>> On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
>>>> wrote:
>>>>>
>>>>>> In this (and the related threads), I see the following three
>>>> requirements:
>>>>>>
>>>>>> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
>>>>>>
>>>>>> 2. "We'll still be releasing 2.x releases for a while, with similar
>>>>>> feature sets as 3.x."
>>>>>>
>>>>>> 3. Avoid the "risk of split-brain behavior" by "minimize
>>> backporting
>>>>>> headaches. Pulling trunk > branch-2 > branch-2.x is already
>>> tedious.
>>>>>> Adding a branch-3, branch-3.x would be obnoxious."
>>>>>>
>>>>>> These three cannot be achieved at the same time.  Which do we
>>> abandon?
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
>>> <sa...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
>>> wrote:
>>>>>>>>
>>>>>>>> 2) Simplification of configs - potentially separating client
>>> side
>>>>>> configs
>>>>>>>> and those used by daemons. This is another source of perpetual
>>>> confusion
>>>>>>>> for users.
>>>>>>> + 1 on this.
>>>>>>>
>>>>>>> sanjay
>>>>>>
>>>>
>>>
>>>
>>>
>

Re: Looking to a Hadoop 3 release

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

+1 for the 3.0 release plan and continuing 2.x releases.
I'm thinking we should consider stopping new 2.x minor releases after 
3.x reaches GA.

Thanks,
Akira

On 2/19/16 10:33, Gangumalla, Uma wrote:
> Yes. I think starting 3.0 release with alpha is good idea. So it would get
> some time to reach the beta or GA.
>
> +1 for the plan.
>
> For the compatibility purposes and as current stable versions, we should
> continue 2.x releases anyway.
>
> Thanks Andrew for starting the thread.
>
> Regards,
> Uma
>
> On 2/18/16, 3:04 PM, "Andrew Wang" <an...@cloudera.com> wrote:
>
>> Hi Kihwal,
>>
>> I think there's still value in continuing the 2.x releases. 3.x comes with
>> the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
>> be beta or GA for some number of months. In the meanwhile, it'd be good to
>> keep putting out regular, stable 2.x releases.
>>
>> Best,
>> Andrew
>>
>>
>> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
>> wrote:
>>
>>> Moving Hadoop 3 forward sounds fine. If EC is one of the main
>>> motivations,
>>> are we getting rid of branch-2.8?
>>>
>>> Kihwal
>>>
>>>        From: Andrew Wang <an...@cloudera.com>
>>>   To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>>> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
>>> hdfs-dev <hd...@hadoop.apache.org>
>>>   Sent: Thursday, February 18, 2016 4:35 PM
>>>   Subject: Re: Looking to a Hadoop 3 release
>>>
>>> Hi all,
>>>
>>> Reviving this thread. I've seen renewed interest in a trunk release
>>> since
>>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
>>> the
>>> shell script rewrite, and many other improvements, I think it's time to
>>> revisit Hadoop 3.0 release plans.
>>>
>>> My overall plan is still the same as in my original email: a series of
>>> regular alpha releases leading up to beta and GA. Alpha releases make it
>>> easier for downstreams to integrate with our code, and making them
>>> regular
>>> means features can be included when they are ready.
>>>
>>> I know there are some incompatible changes waiting in the wings
>>> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
>>> HADOOP-9991 bumping dependency versions) that would be good to get in.
>>> If
>>> you have changes like this, please set the target version to 3.0.0 and
>>> mark
>>> them "Incompatible". We can use this JIRA query to track:
>>>
>>>
>>>
>>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HD
>>> FS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%
>>> 223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flag
>>> s%22%3D%22Incompatible%20change%22%20order%20by%20priority
>>>
>>> There's some release-related stuff that needs to be sorted out (namely,
>>> the
>>> new CHANGES.txt and release note generation from Yetus), but I'd
>>> tentatively like to roll the first alpha a month out, so third week of
>>> March.
>>>
>>> Best,
>>> Andrew
>>>
>>> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com>
>>> wrote:
>>>
>>>> Avoiding the use of JDK8 language features (and, presumably, APIs)
>>>> means you've abandoned #1, i.e., you haven't (really) bumped the JDK
>>>> source version to JDK8.
>>>>
>>>> Also, note that releasing from trunk is a way of achieving #3, it's
>>>> not a way of abandoning it.
>>>>
>>>>
>>>>
>>>> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>>> Hi Raymie,
>>>>>
>>>>> Konst proposed just releasing off of trunk rather than cutting a
>>>> branch-2,
>>>>> and there was general agreement there. So, consider #3 abandoned.
>>> 1&2
>>> can
>>>>> be achieved at the same time, we just need to avoid using JDK8
>>> language
>>>>> features in trunk so things can be backported.
>>>>>
>>>>> Best,
>>>>> Andrew
>>>>>
>>>>> On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
>>>> wrote:
>>>>>
>>>>>> In this (and the related threads), I see the following three
>>>> requirements:
>>>>>>
>>>>>> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
>>>>>>
>>>>>> 2. "We'll still be releasing 2.x releases for a while, with similar
>>>>>> feature sets as 3.x."
>>>>>>
>>>>>> 3. Avoid the "risk of split-brain behavior" by "minimize
>>> backporting
>>>>>> headaches. Pulling trunk > branch-2 > branch-2.x is already
>>> tedious.
>>>>>> Adding a branch-3, branch-3.x would be obnoxious."
>>>>>>
>>>>>> These three cannot be achieved at the same time.  Which do we
>>> abandon?
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
>>> <sa...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
>>> wrote:
>>>>>>>>
>>>>>>>> 2) Simplification of configs - potentially separating client
>>> side
>>>>>> configs
>>>>>>>> and those used by daemons. This is another source of perpetual
>>>> confusion
>>>>>>>> for users.
>>>>>>> + 1 on this.
>>>>>>>
>>>>>>> sanjay
>>>>>>
>>>>
>>>
>>>
>>>
>

Re: Looking to a Hadoop 3 release

["Gangumalla, Uma" <um...@intel.com>]

Yes. I think starting 3.0 release with alpha is good idea. So it would get
some time to reach the beta or GA.

+1 for the plan.

For the compatibility purposes and as current stable versions, we should
continue 2.x releases anyway.

Thanks Andrew for starting the thread.

Regards,
Uma

On 2/18/16, 3:04 PM, "Andrew Wang" <an...@cloudera.com> wrote:

>Hi Kihwal,
>
>I think there's still value in continuing the 2.x releases. 3.x comes with
>the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
>be beta or GA for some number of months. In the meanwhile, it'd be good to
>keep putting out regular, stable 2.x releases.
>
>Best,
>Andrew
>
>
>On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
>wrote:
>
>> Moving Hadoop 3 forward sounds fine. If EC is one of the main
>>motivations,
>> are we getting rid of branch-2.8?
>>
>> Kihwal
>>
>>       From: Andrew Wang <an...@cloudera.com>
>>  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
>> hdfs-dev <hd...@hadoop.apache.org>
>>  Sent: Thursday, February 18, 2016 4:35 PM
>>  Subject: Re: Looking to a Hadoop 3 release
>>
>> Hi all,
>>
>> Reviving this thread. I've seen renewed interest in a trunk release
>>since
>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
>>the
>> shell script rewrite, and many other improvements, I think it's time to
>> revisit Hadoop 3.0 release plans.
>>
>> My overall plan is still the same as in my original email: a series of
>> regular alpha releases leading up to beta and GA. Alpha releases make it
>> easier for downstreams to integrate with our code, and making them
>>regular
>> means features can be included when they are ready.
>>
>> I know there are some incompatible changes waiting in the wings
>> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
>> HADOOP-9991 bumping dependency versions) that would be good to get in.
>>If
>> you have changes like this, please set the target version to 3.0.0 and
>>mark
>> them "Incompatible". We can use this JIRA query to track:
>>
>>
>> 
>>https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HD
>>FS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%
>>223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flag
>>s%22%3D%22Incompatible%20change%22%20order%20by%20priority
>>
>> There's some release-related stuff that needs to be sorted out (namely,
>>the
>> new CHANGES.txt and release note generation from Yetus), but I'd
>> tentatively like to roll the first alpha a month out, so third week of
>> March.
>>
>> Best,
>> Andrew
>>
>> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com>
>>wrote:
>>
>> > Avoiding the use of JDK8 language features (and, presumably, APIs)
>> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
>> > source version to JDK8.
>> >
>> > Also, note that releasing from trunk is a way of achieving #3, it's
>> > not a way of abandoning it.
>> >
>> >
>> >
>> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
>> > wrote:
>> > > Hi Raymie,
>> > >
>> > > Konst proposed just releasing off of trunk rather than cutting a
>> > branch-2,
>> > > and there was general agreement there. So, consider #3 abandoned.
>>1&2
>> can
>> > > be achieved at the same time, we just need to avoid using JDK8
>>language
>> > > features in trunk so things can be backported.
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
>> > wrote:
>> > >
>> > >> In this (and the related threads), I see the following three
>> > requirements:
>> > >>
>> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
>> > >>
>> > >> 2. "We'll still be releasing 2.x releases for a while, with similar
>> > >> feature sets as 3.x."
>> > >>
>> > >> 3. Avoid the "risk of split-brain behavior" by "minimize
>>backporting
>> > >> headaches. Pulling trunk > branch-2 > branch-2.x is already
>>tedious.
>> > >> Adding a branch-3, branch-3.x would be obnoxious."
>> > >>
>> > >> These three cannot be achieved at the same time.  Which do we
>>abandon?
>> > >>
>> > >>
>> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
>><sa...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
>> wrote:
>> > >> >>
>> > >> >> 2) Simplification of configs - potentially separating client
>>side
>> > >> configs
>> > >> >> and those used by daemons. This is another source of perpetual
>> > confusion
>> > >> >> for users.
>> > >> > + 1 on this.
>> > >> >
>> > >> > sanjay
>> > >>
>> >
>>
>>
>>

RE: Looking to a Hadoop 3 release

["Zheng, Kai" <ka...@intel.com>]

Thanks Andrew for driving this. Wonder if it's a good chance for HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note it's not an incompatible change, but feel better to be done in the major release.

Regards,
Kai

-----Original Message-----
From: Andrew Wang [mailto:andrew.wang@cloudera.com] 
Sent: Friday, February 19, 2016 7:04 AM
To: hdfs-dev@hadoop.apache.org; Kihwal Lee <ki...@yahoo-inc.com>
Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release

Hi Kihwal,

I think there's still value in continuing the 2.x releases. 3.x comes with the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't be beta or GA for some number of months. In the meanwhile, it'd be good to keep putting out regular, stable 2.x releases.

Best,
Andrew


On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:

> Moving Hadoop 3 forward sounds fine. If EC is one of the main 
> motivations, are we getting rid of branch-2.8?
>
> Kihwal
>
>       From: Andrew Wang <an...@cloudera.com>
>  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> hdfs-dev <hd...@hadoop.apache.org>
>  Sent: Thursday, February 18, 2016 4:35 PM
>  Subject: Re: Looking to a Hadoop 3 release
>
> Hi all,
>
> Reviving this thread. I've seen renewed interest in a trunk release 
> since HDFS erasure coding has not yet made it to branch-2. Along with 
> JDK8, the shell script rewrite, and many other improvements, I think 
> it's time to revisit Hadoop 3.0 release plans.
>
> My overall plan is still the same as in my original email: a series of 
> regular alpha releases leading up to beta and GA. Alpha releases make 
> it easier for downstreams to integrate with our code, and making them 
> regular means features can be included when they are ready.
>
> I know there are some incompatible changes waiting in the wings (i.e. 
> HDFS-6984 making FileStatus a PB rather than Writable, some of
> HADOOP-9991 bumping dependency versions) that would be good to get in. 
> If you have changes like this, please set the target version to 3.0.0 
> and mark them "Incompatible". We can use this JIRA query to track:
>
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
>
> There's some release-related stuff that needs to be sorted out 
> (namely, the new CHANGES.txt and release note generation from Yetus), 
> but I'd tentatively like to roll the first alpha a month out, so third 
> week of March.
>
> Best,
> Andrew
>
> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:
>
> > Avoiding the use of JDK8 language features (and, presumably, APIs) 
> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK 
> > source version to JDK8.
> >
> > Also, note that releasing from trunk is a way of achieving #3, it's 
> > not a way of abandoning it.
> >
> >
> >
> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang 
> > <an...@cloudera.com>
> > wrote:
> > > Hi Raymie,
> > >
> > > Konst proposed just releasing off of trunk rather than cutting a
> > branch-2,
> > > and there was general agreement there. So, consider #3 abandoned. 
> > > 1&2
> can
> > > be achieved at the same time, we just need to avoid using JDK8 
> > > language features in trunk so things can be backported.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata 
> > > <rs...@altiscale.com>
> > wrote:
> > >
> > >> In this (and the related threads), I see the following three
> > requirements:
> > >>
> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > >>
> > >> 2. "We'll still be releasing 2.x releases for a while, with 
> > >> similar feature sets as 3.x."
> > >>
> > >> 3. Avoid the "risk of split-brain behavior" by "minimize 
> > >> backporting headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> > >> Adding a branch-3, branch-3.x would be obnoxious."
> > >>
> > >> These three cannot be achieved at the same time.  Which do we abandon?
> > >>
> > >>
> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia 
> > >> <sa...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> > >> >>
> > >> >> 2) Simplification of configs - potentially separating client 
> > >> >> side
> > >> configs
> > >> >> and those used by daemons. This is another source of perpetual
> > confusion
> > >> >> for users.
> > >> > + 1 on this.
> > >> >
> > >> > sanjay
> > >>
> >
>
>
>

Re: Looking to a Hadoop 3 release

["Gangumalla, Uma" <um...@intel.com>]

Yes. I think starting 3.0 release with alpha is good idea. So it would get
some time to reach the beta or GA.

+1 for the plan.

For the compatibility purposes and as current stable versions, we should
continue 2.x releases anyway.

Thanks Andrew for starting the thread.

Regards,
Uma

On 2/18/16, 3:04 PM, "Andrew Wang" <an...@cloudera.com> wrote:

>Hi Kihwal,
>
>I think there's still value in continuing the 2.x releases. 3.x comes with
>the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
>be beta or GA for some number of months. In the meanwhile, it'd be good to
>keep putting out regular, stable 2.x releases.
>
>Best,
>Andrew
>
>
>On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
>wrote:
>
>> Moving Hadoop 3 forward sounds fine. If EC is one of the main
>>motivations,
>> are we getting rid of branch-2.8?
>>
>> Kihwal
>>
>>       From: Andrew Wang <an...@cloudera.com>
>>  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
>> hdfs-dev <hd...@hadoop.apache.org>
>>  Sent: Thursday, February 18, 2016 4:35 PM
>>  Subject: Re: Looking to a Hadoop 3 release
>>
>> Hi all,
>>
>> Reviving this thread. I've seen renewed interest in a trunk release
>>since
>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
>>the
>> shell script rewrite, and many other improvements, I think it's time to
>> revisit Hadoop 3.0 release plans.
>>
>> My overall plan is still the same as in my original email: a series of
>> regular alpha releases leading up to beta and GA. Alpha releases make it
>> easier for downstreams to integrate with our code, and making them
>>regular
>> means features can be included when they are ready.
>>
>> I know there are some incompatible changes waiting in the wings
>> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
>> HADOOP-9991 bumping dependency versions) that would be good to get in.
>>If
>> you have changes like this, please set the target version to 3.0.0 and
>>mark
>> them "Incompatible". We can use this JIRA query to track:
>>
>>
>> 
>>https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HD
>>FS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%
>>223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flag
>>s%22%3D%22Incompatible%20change%22%20order%20by%20priority
>>
>> There's some release-related stuff that needs to be sorted out (namely,
>>the
>> new CHANGES.txt and release note generation from Yetus), but I'd
>> tentatively like to roll the first alpha a month out, so third week of
>> March.
>>
>> Best,
>> Andrew
>>
>> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com>
>>wrote:
>>
>> > Avoiding the use of JDK8 language features (and, presumably, APIs)
>> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
>> > source version to JDK8.
>> >
>> > Also, note that releasing from trunk is a way of achieving #3, it's
>> > not a way of abandoning it.
>> >
>> >
>> >
>> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
>> > wrote:
>> > > Hi Raymie,
>> > >
>> > > Konst proposed just releasing off of trunk rather than cutting a
>> > branch-2,
>> > > and there was general agreement there. So, consider #3 abandoned.
>>1&2
>> can
>> > > be achieved at the same time, we just need to avoid using JDK8
>>language
>> > > features in trunk so things can be backported.
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
>> > wrote:
>> > >
>> > >> In this (and the related threads), I see the following three
>> > requirements:
>> > >>
>> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
>> > >>
>> > >> 2. "We'll still be releasing 2.x releases for a while, with similar
>> > >> feature sets as 3.x."
>> > >>
>> > >> 3. Avoid the "risk of split-brain behavior" by "minimize
>>backporting
>> > >> headaches. Pulling trunk > branch-2 > branch-2.x is already
>>tedious.
>> > >> Adding a branch-3, branch-3.x would be obnoxious."
>> > >>
>> > >> These three cannot be achieved at the same time.  Which do we
>>abandon?
>> > >>
>> > >>
>> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
>><sa...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
>> wrote:
>> > >> >>
>> > >> >> 2) Simplification of configs - potentially separating client
>>side
>> > >> configs
>> > >> >> and those used by daemons. This is another source of perpetual
>> > confusion
>> > >> >> for users.
>> > >> > + 1 on this.
>> > >> >
>> > >> > sanjay
>> > >>
>> >
>>
>>
>>

Re: Looking to a Hadoop 3 release

[Andrew Wang <an...@cloudera.com>]

Hi Kihwal,

I think there's still value in continuing the 2.x releases. 3.x comes with
the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
be beta or GA for some number of months. In the meanwhile, it'd be good to
keep putting out regular, stable 2.x releases.

Best,
Andrew


On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:

> Moving Hadoop 3 forward sounds fine. If EC is one of the main motivations,
> are we getting rid of branch-2.8?
>
> Kihwal
>
>       From: Andrew Wang <an...@cloudera.com>
>  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> hdfs-dev <hd...@hadoop.apache.org>
>  Sent: Thursday, February 18, 2016 4:35 PM
>  Subject: Re: Looking to a Hadoop 3 release
>
> Hi all,
>
> Reviving this thread. I've seen renewed interest in a trunk release since
> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
> shell script rewrite, and many other improvements, I think it's time to
> revisit Hadoop 3.0 release plans.
>
> My overall plan is still the same as in my original email: a series of
> regular alpha releases leading up to beta and GA. Alpha releases make it
> easier for downstreams to integrate with our code, and making them regular
> means features can be included when they are ready.
>
> I know there are some incompatible changes waiting in the wings
> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
> HADOOP-9991 bumping dependency versions) that would be good to get in. If
> you have changes like this, please set the target version to 3.0.0 and mark
> them "Incompatible". We can use this JIRA query to track:
>
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
>
> There's some release-related stuff that needs to be sorted out (namely, the
> new CHANGES.txt and release note generation from Yetus), but I'd
> tentatively like to roll the first alpha a month out, so third week of
> March.
>
> Best,
> Andrew
>
> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:
>
> > Avoiding the use of JDK8 language features (and, presumably, APIs)
> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> > source version to JDK8.
> >
> > Also, note that releasing from trunk is a way of achieving #3, it's
> > not a way of abandoning it.
> >
> >
> >
> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
> > wrote:
> > > Hi Raymie,
> > >
> > > Konst proposed just releasing off of trunk rather than cutting a
> > branch-2,
> > > and there was general agreement there. So, consider #3 abandoned. 1&2
> can
> > > be achieved at the same time, we just need to avoid using JDK8 language
> > > features in trunk so things can be backported.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
> > wrote:
> > >
> > >> In this (and the related threads), I see the following three
> > requirements:
> > >>
> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > >>
> > >> 2. "We'll still be releasing 2.x releases for a while, with similar
> > >> feature sets as 3.x."
> > >>
> > >> 3. Avoid the "risk of split-brain behavior" by "minimize backporting
> > >> headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> > >> Adding a branch-3, branch-3.x would be obnoxious."
> > >>
> > >> These three cannot be achieved at the same time.  Which do we abandon?
> > >>
> > >>
> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia <sa...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> > >> >>
> > >> >> 2) Simplification of configs - potentially separating client side
> > >> configs
> > >> >> and those used by daemons. This is another source of perpetual
> > confusion
> > >> >> for users.
> > >> > + 1 on this.
> > >> >
> > >> > sanjay
> > >>
> >
>
>
>

Re: Looking to a Hadoop 3 release

[Andrew Wang <an...@cloudera.com>]

Hi Kihwal,

I think there's still value in continuing the 2.x releases. 3.x comes with
the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
be beta or GA for some number of months. In the meanwhile, it'd be good to
keep putting out regular, stable 2.x releases.

Best,
Andrew


On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:

> Moving Hadoop 3 forward sounds fine. If EC is one of the main motivations,
> are we getting rid of branch-2.8?
>
> Kihwal
>
>       From: Andrew Wang <an...@cloudera.com>
>  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> hdfs-dev <hd...@hadoop.apache.org>
>  Sent: Thursday, February 18, 2016 4:35 PM
>  Subject: Re: Looking to a Hadoop 3 release
>
> Hi all,
>
> Reviving this thread. I've seen renewed interest in a trunk release since
> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
> shell script rewrite, and many other improvements, I think it's time to
> revisit Hadoop 3.0 release plans.
>
> My overall plan is still the same as in my original email: a series of
> regular alpha releases leading up to beta and GA. Alpha releases make it
> easier for downstreams to integrate with our code, and making them regular
> means features can be included when they are ready.
>
> I know there are some incompatible changes waiting in the wings
> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
> HADOOP-9991 bumping dependency versions) that would be good to get in. If
> you have changes like this, please set the target version to 3.0.0 and mark
> them "Incompatible". We can use this JIRA query to track:
>
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
>
> There's some release-related stuff that needs to be sorted out (namely, the
> new CHANGES.txt and release note generation from Yetus), but I'd
> tentatively like to roll the first alpha a month out, so third week of
> March.
>
> Best,
> Andrew
>
> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:
>
> > Avoiding the use of JDK8 language features (and, presumably, APIs)
> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> > source version to JDK8.
> >
> > Also, note that releasing from trunk is a way of achieving #3, it's
> > not a way of abandoning it.
> >
> >
> >
> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
> > wrote:
> > > Hi Raymie,
> > >
> > > Konst proposed just releasing off of trunk rather than cutting a
> > branch-2,
> > > and there was general agreement there. So, consider #3 abandoned. 1&2
> can
> > > be achieved at the same time, we just need to avoid using JDK8 language
> > > features in trunk so things can be backported.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
> > wrote:
> > >
> > >> In this (and the related threads), I see the following three
> > requirements:
> > >>
> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > >>
> > >> 2. "We'll still be releasing 2.x releases for a while, with similar
> > >> feature sets as 3.x."
> > >>
> > >> 3. Avoid the "risk of split-brain behavior" by "minimize backporting
> > >> headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> > >> Adding a branch-3, branch-3.x would be obnoxious."
> > >>
> > >> These three cannot be achieved at the same time.  Which do we abandon?
> > >>
> > >>
> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia <sa...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> > >> >>
> > >> >> 2) Simplification of configs - potentially separating client side
> > >> configs
> > >> >> and those used by daemons. This is another source of perpetual
> > confusion
> > >> >> for users.
> > >> > + 1 on this.
> > >> >
> > >> > sanjay
> > >>
> >
>
>
>

Re: Looking to a Hadoop 3 release

[Andrew Wang <an...@cloudera.com>]

Hi Kihwal,

I think there's still value in continuing the 2.x releases. 3.x comes with
the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
be beta or GA for some number of months. In the meanwhile, it'd be good to
keep putting out regular, stable 2.x releases.

Best,
Andrew


On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:

> Moving Hadoop 3 forward sounds fine. If EC is one of the main motivations,
> are we getting rid of branch-2.8?
>
> Kihwal
>
>       From: Andrew Wang <an...@cloudera.com>
>  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> hdfs-dev <hd...@hadoop.apache.org>
>  Sent: Thursday, February 18, 2016 4:35 PM
>  Subject: Re: Looking to a Hadoop 3 release
>
> Hi all,
>
> Reviving this thread. I've seen renewed interest in a trunk release since
> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
> shell script rewrite, and many other improvements, I think it's time to
> revisit Hadoop 3.0 release plans.
>
> My overall plan is still the same as in my original email: a series of
> regular alpha releases leading up to beta and GA. Alpha releases make it
> easier for downstreams to integrate with our code, and making them regular
> means features can be included when they are ready.
>
> I know there are some incompatible changes waiting in the wings
> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
> HADOOP-9991 bumping dependency versions) that would be good to get in. If
> you have changes like this, please set the target version to 3.0.0 and mark
> them "Incompatible". We can use this JIRA query to track:
>
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
>
> There's some release-related stuff that needs to be sorted out (namely, the
> new CHANGES.txt and release note generation from Yetus), but I'd
> tentatively like to roll the first alpha a month out, so third week of
> March.
>
> Best,
> Andrew
>
> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:
>
> > Avoiding the use of JDK8 language features (and, presumably, APIs)
> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> > source version to JDK8.
> >
> > Also, note that releasing from trunk is a way of achieving #3, it's
> > not a way of abandoning it.
> >
> >
> >
> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
> > wrote:
> > > Hi Raymie,
> > >
> > > Konst proposed just releasing off of trunk rather than cutting a
> > branch-2,
> > > and there was general agreement there. So, consider #3 abandoned. 1&2
> can
> > > be achieved at the same time, we just need to avoid using JDK8 language
> > > features in trunk so things can be backported.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
> > wrote:
> > >
> > >> In this (and the related threads), I see the following three
> > requirements:
> > >>
> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > >>
> > >> 2. "We'll still be releasing 2.x releases for a while, with similar
> > >> feature sets as 3.x."
> > >>
> > >> 3. Avoid the "risk of split-brain behavior" by "minimize backporting
> > >> headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> > >> Adding a branch-3, branch-3.x would be obnoxious."
> > >>
> > >> These three cannot be achieved at the same time.  Which do we abandon?
> > >>
> > >>
> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia <sa...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> > >> >>
> > >> >> 2) Simplification of configs - potentially separating client side
> > >> configs
> > >> >> and those used by daemons. This is another source of perpetual
> > confusion
> > >> >> for users.
> > >> > + 1 on this.
> > >> >
> > >> > sanjay
> > >>
> >
>
>
>

Re: Looking to a Hadoop 3 release

[Andrew Wang <an...@cloudera.com>]

Hi Kihwal,

I think there's still value in continuing the 2.x releases. 3.x comes with
the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
be beta or GA for some number of months. In the meanwhile, it'd be good to
keep putting out regular, stable 2.x releases.

Best,
Andrew


On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:

> Moving Hadoop 3 forward sounds fine. If EC is one of the main motivations,
> are we getting rid of branch-2.8?
>
> Kihwal
>
>       From: Andrew Wang <an...@cloudera.com>
>  To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>;
> hdfs-dev <hd...@hadoop.apache.org>
>  Sent: Thursday, February 18, 2016 4:35 PM
>  Subject: Re: Looking to a Hadoop 3 release
>
> Hi all,
>
> Reviving this thread. I've seen renewed interest in a trunk release since
> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
> shell script rewrite, and many other improvements, I think it's time to
> revisit Hadoop 3.0 release plans.
>
> My overall plan is still the same as in my original email: a series of
> regular alpha releases leading up to beta and GA. Alpha releases make it
> easier for downstreams to integrate with our code, and making them regular
> means features can be included when they are ready.
>
> I know there are some incompatible changes waiting in the wings
> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
> HADOOP-9991 bumping dependency versions) that would be good to get in. If
> you have changes like this, please set the target version to 3.0.0 and mark
> them "Incompatible". We can use this JIRA query to track:
>
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
>
> There's some release-related stuff that needs to be sorted out (namely, the
> new CHANGES.txt and release note generation from Yetus), but I'd
> tentatively like to roll the first alpha a month out, so third week of
> March.
>
> Best,
> Andrew
>
> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:
>
> > Avoiding the use of JDK8 language features (and, presumably, APIs)
> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> > source version to JDK8.
> >
> > Also, note that releasing from trunk is a way of achieving #3, it's
> > not a way of abandoning it.
> >
> >
> >
> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
> > wrote:
> > > Hi Raymie,
> > >
> > > Konst proposed just releasing off of trunk rather than cutting a
> > branch-2,
> > > and there was general agreement there. So, consider #3 abandoned. 1&2
> can
> > > be achieved at the same time, we just need to avoid using JDK8 language
> > > features in trunk so things can be backported.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
> > wrote:
> > >
> > >> In this (and the related threads), I see the following three
> > requirements:
> > >>
> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > >>
> > >> 2. "We'll still be releasing 2.x releases for a while, with similar
> > >> feature sets as 3.x."
> > >>
> > >> 3. Avoid the "risk of split-brain behavior" by "minimize backporting
> > >> headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> > >> Adding a branch-3, branch-3.x would be obnoxious."
> > >>
> > >> These three cannot be achieved at the same time.  Which do we abandon?
> > >>
> > >>
> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia <sa...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> > >> >>
> > >> >> 2) Simplification of configs - potentially separating client side
> > >> configs
> > >> >> and those used by daemons. This is another source of perpetual
> > confusion
> > >> >> for users.
> > >> > + 1 on this.
> > >> >
> > >> > sanjay
> > >>
> >
>
>
>

Re: Looking to a Hadoop 3 release

[Kihwal Lee <ki...@yahoo-inc.com.INVALID>]

Moving Hadoop 3 forward sounds fine. If EC is one of the main motivations, are we getting rid of branch-2.8? 

Kihwal

      From: Andrew Wang <an...@cloudera.com>
 To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org> 
Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
 Sent: Thursday, February 18, 2016 4:35 PM
 Subject: Re: Looking to a Hadoop 3 release
   
Hi all,

Reviving this thread. I've seen renewed interest in a trunk release since
HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
shell script rewrite, and many other improvements, I think it's time to
revisit Hadoop 3.0 release plans.

My overall plan is still the same as in my original email: a series of
regular alpha releases leading up to beta and GA. Alpha releases make it
easier for downstreams to integrate with our code, and making them regular
means features can be included when they are ready.

I know there are some incompatible changes waiting in the wings
(i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
HADOOP-9991 bumping dependency versions) that would be good to get in. If
you have changes like this, please set the target version to 3.0.0 and mark
them "Incompatible". We can use this JIRA query to track:

https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority

There's some release-related stuff that needs to be sorted out (namely, the
new CHANGES.txt and release note generation from Yetus), but I'd
tentatively like to roll the first alpha a month out, so third week of
March.

Best,
Andrew

On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:

> Avoiding the use of JDK8 language features (and, presumably, APIs)
> means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> source version to JDK8.
>
> Also, note that releasing from trunk is a way of achieving #3, it's
> not a way of abandoning it.
>
>
>
> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> > Hi Raymie,
> >
> > Konst proposed just releasing off of trunk rather than cutting a
> branch-2,
> > and there was general agreement there. So, consider #3 abandoned. 1&2 can
> > be achieved at the same time, we just need to avoid using JDK8 language
> > features in trunk so things can be backported.
> >
> > Best,
> > Andrew
> >
> > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
> wrote:
> >
> >> In this (and the related threads), I see the following three
> requirements:
> >>
> >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> >>
> >> 2. "We'll still be releasing 2.x releases for a while, with similar
> >> feature sets as 3.x."
> >>
> >> 3. Avoid the "risk of split-brain behavior" by "minimize backporting
> >> headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> >> Adding a branch-3, branch-3.x would be obnoxious."
> >>
> >> These three cannot be achieved at the same time.  Which do we abandon?
> >>
> >>
> >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia <sa...@gmail.com>
> >> wrote:
> >> >
> >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org> wrote:
> >> >>
> >> >> 2) Simplification of configs - potentially separating client side
> >> configs
> >> >> and those used by daemons. This is another source of perpetual
> confusion
> >> >> for users.
> >> > + 1 on this.
> >> >
> >> > sanjay
> >>
>

  

Re: Looking to a Hadoop 3 release

[Kihwal Lee <ki...@yahoo-inc.com.INVALID>]

Moving Hadoop 3 forward sounds fine. If EC is one of the main motivations, are we getting rid of branch-2.8? 

Kihwal

      From: Andrew Wang <an...@cloudera.com>
 To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org> 
Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
 Sent: Thursday, February 18, 2016 4:35 PM
 Subject: Re: Looking to a Hadoop 3 release
   
Hi all,

Reviving this thread. I've seen renewed interest in a trunk release since
HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
shell script rewrite, and many other improvements, I think it's time to
revisit Hadoop 3.0 release plans.

My overall plan is still the same as in my original email: a series of
regular alpha releases leading up to beta and GA. Alpha releases make it
easier for downstreams to integrate with our code, and making them regular
means features can be included when they are ready.

I know there are some incompatible changes waiting in the wings
(i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
HADOOP-9991 bumping dependency versions) that would be good to get in. If
you have changes like this, please set the target version to 3.0.0 and mark
them "Incompatible". We can use this JIRA query to track:

https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority

There's some release-related stuff that needs to be sorted out (namely, the
new CHANGES.txt and release note generation from Yetus), but I'd
tentatively like to roll the first alpha a month out, so third week of
March.

Best,
Andrew

On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:

> Avoiding the use of JDK8 language features (and, presumably, APIs)
> means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> source version to JDK8.
>
> Also, note that releasing from trunk is a way of achieving #3, it's
> not a way of abandoning it.
>
>
>
> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> > Hi Raymie,
> >
> > Konst proposed just releasing off of trunk rather than cutting a
> branch-2,
> > and there was general agreement there. So, consider #3 abandoned. 1&2 can
> > be achieved at the same time, we just need to avoid using JDK8 language
> > features in trunk so things can be backported.
> >
> > Best,
> > Andrew
> >
> > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
> wrote:
> >
> >> In this (and the related threads), I see the following three
> requirements:
> >>
> >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> >>
> >> 2. "We'll still be releasing 2.x releases for a while, with similar
> >> feature sets as 3.x."
> >>
> >> 3. Avoid the "risk of split-brain behavior" by "minimize backporting
> >> headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> >> Adding a branch-3, branch-3.x would be obnoxious."
> >>
> >> These three cannot be achieved at the same time.  Which do we abandon?
> >>
> >>
> >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia <sa...@gmail.com>
> >> wrote:
> >> >
> >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org> wrote:
> >> >>
> >> >> 2) Simplification of configs - potentially separating client side
> >> configs
> >> >> and those used by daemons. This is another source of perpetual
> confusion
> >> >> for users.
> >> > + 1 on this.
> >> >
> >> > sanjay
> >>
>

  

Re: Looking to a Hadoop 3 release

[Kihwal Lee <ki...@yahoo-inc.com.INVALID>]

Moving Hadoop 3 forward sounds fine. If EC is one of the main motivations, are we getting rid of branch-2.8? 

Kihwal

      From: Andrew Wang <an...@cloudera.com>
 To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org> 
Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
 Sent: Thursday, February 18, 2016 4:35 PM
 Subject: Re: Looking to a Hadoop 3 release
   
Hi all,

Reviving this thread. I've seen renewed interest in a trunk release since
HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
shell script rewrite, and many other improvements, I think it's time to
revisit Hadoop 3.0 release plans.

My overall plan is still the same as in my original email: a series of
regular alpha releases leading up to beta and GA. Alpha releases make it
easier for downstreams to integrate with our code, and making them regular
means features can be included when they are ready.

I know there are some incompatible changes waiting in the wings
(i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
HADOOP-9991 bumping dependency versions) that would be good to get in. If
you have changes like this, please set the target version to 3.0.0 and mark
them "Incompatible". We can use this JIRA query to track:

https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority

There's some release-related stuff that needs to be sorted out (namely, the
new CHANGES.txt and release note generation from Yetus), but I'd
tentatively like to roll the first alpha a month out, so third week of
March.

Best,
Andrew

On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:

> Avoiding the use of JDK8 language features (and, presumably, APIs)
> means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> source version to JDK8.
>
> Also, note that releasing from trunk is a way of achieving #3, it's
> not a way of abandoning it.
>
>
>
> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> > Hi Raymie,
> >
> > Konst proposed just releasing off of trunk rather than cutting a
> branch-2,
> > and there was general agreement there. So, consider #3 abandoned. 1&2 can
> > be achieved at the same time, we just need to avoid using JDK8 language
> > features in trunk so things can be backported.
> >
> > Best,
> > Andrew
> >
> > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
> wrote:
> >
> >> In this (and the related threads), I see the following three
> requirements:
> >>
> >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> >>
> >> 2. "We'll still be releasing 2.x releases for a while, with similar
> >> feature sets as 3.x."
> >>
> >> 3. Avoid the "risk of split-brain behavior" by "minimize backporting
> >> headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> >> Adding a branch-3, branch-3.x would be obnoxious."
> >>
> >> These three cannot be achieved at the same time.  Which do we abandon?
> >>
> >>
> >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia <sa...@gmail.com>
> >> wrote:
> >> >
> >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org> wrote:
> >> >>
> >> >> 2) Simplification of configs - potentially separating client side
> >> configs
> >> >> and those used by daemons. This is another source of perpetual
> confusion
> >> >> for users.
> >> > + 1 on this.
> >> >
> >> > sanjay
> >>
>

  

Re: Looking to a Hadoop 3 release

[Kihwal Lee <ki...@yahoo-inc.com.INVALID>]

Moving Hadoop 3 forward sounds fine. If EC is one of the main motivations, are we getting rid of branch-2.8? 

Kihwal

      From: Andrew Wang <an...@cloudera.com>
 To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org> 
Cc: "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
 Sent: Thursday, February 18, 2016 4:35 PM
 Subject: Re: Looking to a Hadoop 3 release
   
Hi all,

Reviving this thread. I've seen renewed interest in a trunk release since
HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
shell script rewrite, and many other improvements, I think it's time to
revisit Hadoop 3.0 release plans.

My overall plan is still the same as in my original email: a series of
regular alpha releases leading up to beta and GA. Alpha releases make it
easier for downstreams to integrate with our code, and making them regular
means features can be included when they are ready.

I know there are some incompatible changes waiting in the wings
(i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
HADOOP-9991 bumping dependency versions) that would be good to get in. If
you have changes like this, please set the target version to 3.0.0 and mark
them "Incompatible". We can use this JIRA query to track:

https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority

There's some release-related stuff that needs to be sorted out (namely, the
new CHANGES.txt and release note generation from Yetus), but I'd
tentatively like to roll the first alpha a month out, so third week of
March.

Best,
Andrew

On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rs...@altiscale.com> wrote:

> Avoiding the use of JDK8 language features (and, presumably, APIs)
> means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> source version to JDK8.
>
> Also, note that releasing from trunk is a way of achieving #3, it's
> not a way of abandoning it.
>
>
>
> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> > Hi Raymie,
> >
> > Konst proposed just releasing off of trunk rather than cutting a
> branch-2,
> > and there was general agreement there. So, consider #3 abandoned. 1&2 can
> > be achieved at the same time, we just need to avoid using JDK8 language
> > features in trunk so things can be backported.
> >
> > Best,
> > Andrew
> >
> > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata <rs...@altiscale.com>
> wrote:
> >
> >> In this (and the related threads), I see the following three
> requirements:
> >>
> >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> >>
> >> 2. "We'll still be releasing 2.x releases for a while, with similar
> >> feature sets as 3.x."
> >>
> >> 3. Avoid the "risk of split-brain behavior" by "minimize backporting
> >> headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> >> Adding a branch-3, branch-3.x would be obnoxious."
> >>
> >> These three cannot be achieved at the same time.  Which do we abandon?
> >>
> >>
> >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia <sa...@gmail.com>
> >> wrote:
> >> >
> >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org> wrote:
> >> >>
> >> >> 2) Simplification of configs - potentially separating client side
> >> configs
> >> >> and those used by daemons. This is another source of perpetual
> confusion
> >> >> for users.
> >> > + 1 on this.
> >> >
> >> > sanjay
> >>
>

  

Re: Looking to a Hadoop 3 release

[Ravi Prakash <ra...@gmail.com>]

+1 for the plan to start cutting 3.x alpha releases. Thanks for the
initiative Andrew!

On Fri, Feb 19, 2016 at 6:19 AM, Steve Loughran <st...@hortonworks.com>
wrote:

>
> > On 19 Feb 2016, at 11:27, Dmitry Sivachenko <tr...@gmail.com> wrote:
> >
> >
> >> On 19 Feb 2016, at 01:35, Andrew Wang <an...@cloudera.com> wrote:
> >>
> >> Hi all,
> >>
> >> Reviving this thread. I've seen renewed interest in a trunk release
> since
> >> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
> the
> >> shell script rewrite, and many other improvements, I think it's time to
> >> revisit Hadoop 3.0 release plans.
> >>
> >
>
> It's time to start ... I suspect it'll take a while to stabilise. I look
> forward to the new shell scripts already
>
> One thing I do want there is for all the alpha releases to make clear that
> there are no compatibility policies here; protocols may change and there is
> no requirement of the first 3.x release to be compatible with all the 3.0.x
> alphas. That's something we missed out on the 2.0.x-alpha process, or at
> least not repeated often enough.
>
> >
> > Hello,
> >
> > any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes
> out?
> >
> > Thanks!
> >
> >
>
> sounds like a good time for a status update on the FB work —and anything
> people can do to test it would be appreciated by all. That includes testing
> on ipv4 systems, and especially, IPv4/v6 systems with Kerberos turned on
> and both MIT and AD kerberos servers. At the same time, IPv6 support ought
> to be something that could be added in.
>
>
> I don't have any opinions on timescale, but
>
> +1 to anything related to classpath isolation
> +1 to a careful bump of versions of dependencies.
> +1 to fixing the outstanding Java 8 migration issues, especially the big
> Jersey patch that's just been updated.
> +1 to switching to JIRA-created release notes
>
> Having been doing the slider releases recently, it's clear to me that you
> can do a lot in automating the release process itself. All those steps in
> the release runbook can be turned into targets in a special ant release.xml
> build file, calling maven, gpg, etc.
>
> I think doing something like this for 3.0 will significantly benefit both
> the release phase here but the future releases
>
> This is the slider one:
> https://github.com/apache/incubator-slider/blob/develop/bin/release.xml
>
> It doesn't replace maven, instead it choreographs that along with all the
> other steps: signing and checksumming artifacts, publishing them, voting
>
> it includes
>  -refusing to release if the git repo is modified
>  -making the various git branch/tag/push operations
>  -issuing the various mvn versions:update commands
>  -signing
>  -publishing via asf SVN
>  -using GET calls too verify the artifacts made it
>  -generating the vote and vote result emails (it even counts the votes)
>
> I recommend this is included as part of the release process. It does make
> a difference; we can now cut new releases with no human intervention other
> than editing a properties file and running different targets as the process
> goes through its release and vote phases.
>
> -Steve

Re: Looking to a Hadoop 3 release

[Ravi Prakash <ra...@gmail.com>]

+1 for the plan to start cutting 3.x alpha releases. Thanks for the
initiative Andrew!

On Fri, Feb 19, 2016 at 6:19 AM, Steve Loughran <st...@hortonworks.com>
wrote:

>
> > On 19 Feb 2016, at 11:27, Dmitry Sivachenko <tr...@gmail.com> wrote:
> >
> >
> >> On 19 Feb 2016, at 01:35, Andrew Wang <an...@cloudera.com> wrote:
> >>
> >> Hi all,
> >>
> >> Reviving this thread. I've seen renewed interest in a trunk release
> since
> >> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
> the
> >> shell script rewrite, and many other improvements, I think it's time to
> >> revisit Hadoop 3.0 release plans.
> >>
> >
>
> It's time to start ... I suspect it'll take a while to stabilise. I look
> forward to the new shell scripts already
>
> One thing I do want there is for all the alpha releases to make clear that
> there are no compatibility policies here; protocols may change and there is
> no requirement of the first 3.x release to be compatible with all the 3.0.x
> alphas. That's something we missed out on the 2.0.x-alpha process, or at
> least not repeated often enough.
>
> >
> > Hello,
> >
> > any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes
> out?
> >
> > Thanks!
> >
> >
>
> sounds like a good time for a status update on the FB work —and anything
> people can do to test it would be appreciated by all. That includes testing
> on ipv4 systems, and especially, IPv4/v6 systems with Kerberos turned on
> and both MIT and AD kerberos servers. At the same time, IPv6 support ought
> to be something that could be added in.
>
>
> I don't have any opinions on timescale, but
>
> +1 to anything related to classpath isolation
> +1 to a careful bump of versions of dependencies.
> +1 to fixing the outstanding Java 8 migration issues, especially the big
> Jersey patch that's just been updated.
> +1 to switching to JIRA-created release notes
>
> Having been doing the slider releases recently, it's clear to me that you
> can do a lot in automating the release process itself. All those steps in
> the release runbook can be turned into targets in a special ant release.xml
> build file, calling maven, gpg, etc.
>
> I think doing something like this for 3.0 will significantly benefit both
> the release phase here but the future releases
>
> This is the slider one:
> https://github.com/apache/incubator-slider/blob/develop/bin/release.xml
>
> It doesn't replace maven, instead it choreographs that along with all the
> other steps: signing and checksumming artifacts, publishing them, voting
>
> it includes
>  -refusing to release if the git repo is modified
>  -making the various git branch/tag/push operations
>  -issuing the various mvn versions:update commands
>  -signing
>  -publishing via asf SVN
>  -using GET calls too verify the artifacts made it
>  -generating the vote and vote result emails (it even counts the votes)
>
> I recommend this is included as part of the release process. It does make
> a difference; we can now cut new releases with no human intervention other
> than editing a properties file and running different targets as the process
> goes through its release and vote phases.
>
> -Steve

Re: Looking to a Hadoop 3 release

[Ravi Prakash <ra...@gmail.com>]

+1 for the plan to start cutting 3.x alpha releases. Thanks for the
initiative Andrew!

On Fri, Feb 19, 2016 at 6:19 AM, Steve Loughran <st...@hortonworks.com>
wrote:

>
> > On 19 Feb 2016, at 11:27, Dmitry Sivachenko <tr...@gmail.com> wrote:
> >
> >
> >> On 19 Feb 2016, at 01:35, Andrew Wang <an...@cloudera.com> wrote:
> >>
> >> Hi all,
> >>
> >> Reviving this thread. I've seen renewed interest in a trunk release
> since
> >> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
> the
> >> shell script rewrite, and many other improvements, I think it's time to
> >> revisit Hadoop 3.0 release plans.
> >>
> >
>
> It's time to start ... I suspect it'll take a while to stabilise. I look
> forward to the new shell scripts already
>
> One thing I do want there is for all the alpha releases to make clear that
> there are no compatibility policies here; protocols may change and there is
> no requirement of the first 3.x release to be compatible with all the 3.0.x
> alphas. That's something we missed out on the 2.0.x-alpha process, or at
> least not repeated often enough.
>
> >
> > Hello,
> >
> > any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes
> out?
> >
> > Thanks!
> >
> >
>
> sounds like a good time for a status update on the FB work —and anything
> people can do to test it would be appreciated by all. That includes testing
> on ipv4 systems, and especially, IPv4/v6 systems with Kerberos turned on
> and both MIT and AD kerberos servers. At the same time, IPv6 support ought
> to be something that could be added in.
>
>
> I don't have any opinions on timescale, but
>
> +1 to anything related to classpath isolation
> +1 to a careful bump of versions of dependencies.
> +1 to fixing the outstanding Java 8 migration issues, especially the big
> Jersey patch that's just been updated.
> +1 to switching to JIRA-created release notes
>
> Having been doing the slider releases recently, it's clear to me that you
> can do a lot in automating the release process itself. All those steps in
> the release runbook can be turned into targets in a special ant release.xml
> build file, calling maven, gpg, etc.
>
> I think doing something like this for 3.0 will significantly benefit both
> the release phase here but the future releases
>
> This is the slider one:
> https://github.com/apache/incubator-slider/blob/develop/bin/release.xml
>
> It doesn't replace maven, instead it choreographs that along with all the
> other steps: signing and checksumming artifacts, publishing them, voting
>
> it includes
>  -refusing to release if the git repo is modified
>  -making the various git branch/tag/push operations
>  -issuing the various mvn versions:update commands
>  -signing
>  -publishing via asf SVN
>  -using GET calls too verify the artifacts made it
>  -generating the vote and vote result emails (it even counts the votes)
>
> I recommend this is included as part of the release process. It does make
> a difference; we can now cut new releases with no human intervention other
> than editing a properties file and running different targets as the process
> goes through its release and vote phases.
>
> -Steve

Re: Looking to a Hadoop 3 release

[Ravi Prakash <ra...@gmail.com>]

+1 for the plan to start cutting 3.x alpha releases. Thanks for the
initiative Andrew!

On Fri, Feb 19, 2016 at 6:19 AM, Steve Loughran <st...@hortonworks.com>
wrote:

>
> > On 19 Feb 2016, at 11:27, Dmitry Sivachenko <tr...@gmail.com> wrote:
> >
> >
> >> On 19 Feb 2016, at 01:35, Andrew Wang <an...@cloudera.com> wrote:
> >>
> >> Hi all,
> >>
> >> Reviving this thread. I've seen renewed interest in a trunk release
> since
> >> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
> the
> >> shell script rewrite, and many other improvements, I think it's time to
> >> revisit Hadoop 3.0 release plans.
> >>
> >
>
> It's time to start ... I suspect it'll take a while to stabilise. I look
> forward to the new shell scripts already
>
> One thing I do want there is for all the alpha releases to make clear that
> there are no compatibility policies here; protocols may change and there is
> no requirement of the first 3.x release to be compatible with all the 3.0.x
> alphas. That's something we missed out on the 2.0.x-alpha process, or at
> least not repeated often enough.
>
> >
> > Hello,
> >
> > any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes
> out?
> >
> > Thanks!
> >
> >
>
> sounds like a good time for a status update on the FB work —and anything
> people can do to test it would be appreciated by all. That includes testing
> on ipv4 systems, and especially, IPv4/v6 systems with Kerberos turned on
> and both MIT and AD kerberos servers. At the same time, IPv6 support ought
> to be something that could be added in.
>
>
> I don't have any opinions on timescale, but
>
> +1 to anything related to classpath isolation
> +1 to a careful bump of versions of dependencies.
> +1 to fixing the outstanding Java 8 migration issues, especially the big
> Jersey patch that's just been updated.
> +1 to switching to JIRA-created release notes
>
> Having been doing the slider releases recently, it's clear to me that you
> can do a lot in automating the release process itself. All those steps in
> the release runbook can be turned into targets in a special ant release.xml
> build file, calling maven, gpg, etc.
>
> I think doing something like this for 3.0 will significantly benefit both
> the release phase here but the future releases
>
> This is the slider one:
> https://github.com/apache/incubator-slider/blob/develop/bin/release.xml
>
> It doesn't replace maven, instead it choreographs that along with all the
> other steps: signing and checksumming artifacts, publishing them, voting
>
> it includes
>  -refusing to release if the git repo is modified
>  -making the various git branch/tag/push operations
>  -issuing the various mvn versions:update commands
>  -signing
>  -publishing via asf SVN
>  -using GET calls too verify the artifacts made it
>  -generating the vote and vote result emails (it even counts the votes)
>
> I recommend this is included as part of the release process. It does make
> a difference; we can now cut new releases with no human intervention other
> than editing a properties file and running different targets as the process
> goes through its release and vote phases.
>
> -Steve

Re: Looking to a Hadoop 3 release

[Steve Loughran <st...@hortonworks.com>]

> On 19 Feb 2016, at 11:27, Dmitry Sivachenko <tr...@gmail.com> wrote:
> 
> 
>> On 19 Feb 2016, at 01:35, Andrew Wang <an...@cloudera.com> wrote:
>> 
>> Hi all,
>> 
>> Reviving this thread. I've seen renewed interest in a trunk release since
>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
>> shell script rewrite, and many other improvements, I think it's time to
>> revisit Hadoop 3.0 release plans.
>> 
> 

It's time to start ... I suspect it'll take a while to stabilise. I look forward to the new shell scripts already

One thing I do want there is for all the alpha releases to make clear that there are no compatibility policies here; protocols may change and there is no requirement of the first 3.x release to be compatible with all the 3.0.x alphas. That's something we missed out on the 2.0.x-alpha process, or at least not repeated often enough.

> 
> Hello,
> 
> any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes out?
> 
> Thanks!
> 
> 

sounds like a good time for a status update on the FB work —and anything people can do to test it would be appreciated by all. That includes testing on ipv4 systems, and especially, IPv4/v6 systems with Kerberos turned on and both MIT and AD kerberos servers. At the same time, IPv6 support ought to be something that could be added in.


I don't have any opinions on timescale, but

+1 to anything related to classpath isolation
+1 to a careful bump of versions of dependencies.
+1 to fixing the outstanding Java 8 migration issues, especially the big Jersey patch that's just been updated.
+1 to switching to JIRA-created release notes

Having been doing the slider releases recently, it's clear to me that you can do a lot in automating the release process itself. All those steps in the release runbook can be turned into targets in a special ant release.xml build file, calling maven, gpg, etc.

I think doing something like this for 3.0 will significantly benefit both the release phase here but the future releases

This is the slider one: https://github.com/apache/incubator-slider/blob/develop/bin/release.xml

It doesn't replace maven, instead it choreographs that along with all the other steps: signing and checksumming artifacts, publishing them, voting

it includes
 -refusing to release if the git repo is modified
 -making the various git branch/tag/push operations
 -issuing the various mvn versions:update commands
 -signing
 -publishing via asf SVN 
 -using GET calls too verify the artifacts made it
 -generating the vote and vote result emails (it even counts the votes)

I recommend this is included as part of the release process. It does make a difference; we can now cut new releases with no human intervention other than editing a properties file and running different targets as the process goes through its release and vote phases.

-Steve

Re: Looking to a Hadoop 3 release

[Steve Loughran <st...@hortonworks.com>]

> On 19 Feb 2016, at 11:27, Dmitry Sivachenko <tr...@gmail.com> wrote:
> 
> 
>> On 19 Feb 2016, at 01:35, Andrew Wang <an...@cloudera.com> wrote:
>> 
>> Hi all,
>> 
>> Reviving this thread. I've seen renewed interest in a trunk release since
>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
>> shell script rewrite, and many other improvements, I think it's time to
>> revisit Hadoop 3.0 release plans.
>> 
> 

It's time to start ... I suspect it'll take a while to stabilise. I look forward to the new shell scripts already

One thing I do want there is for all the alpha releases to make clear that there are no compatibility policies here; protocols may change and there is no requirement of the first 3.x release to be compatible with all the 3.0.x alphas. That's something we missed out on the 2.0.x-alpha process, or at least not repeated often enough.

> 
> Hello,
> 
> any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes out?
> 
> Thanks!
> 
> 

sounds like a good time for a status update on the FB work —and anything people can do to test it would be appreciated by all. That includes testing on ipv4 systems, and especially, IPv4/v6 systems with Kerberos turned on and both MIT and AD kerberos servers. At the same time, IPv6 support ought to be something that could be added in.


I don't have any opinions on timescale, but

+1 to anything related to classpath isolation
+1 to a careful bump of versions of dependencies.
+1 to fixing the outstanding Java 8 migration issues, especially the big Jersey patch that's just been updated.
+1 to switching to JIRA-created release notes

Having been doing the slider releases recently, it's clear to me that you can do a lot in automating the release process itself. All those steps in the release runbook can be turned into targets in a special ant release.xml build file, calling maven, gpg, etc.

I think doing something like this for 3.0 will significantly benefit both the release phase here but the future releases

This is the slider one: https://github.com/apache/incubator-slider/blob/develop/bin/release.xml

It doesn't replace maven, instead it choreographs that along with all the other steps: signing and checksumming artifacts, publishing them, voting

it includes
 -refusing to release if the git repo is modified
 -making the various git branch/tag/push operations
 -issuing the various mvn versions:update commands
 -signing
 -publishing via asf SVN 
 -using GET calls too verify the artifacts made it
 -generating the vote and vote result emails (it even counts the votes)

I recommend this is included as part of the release process. It does make a difference; we can now cut new releases with no human intervention other than editing a properties file and running different targets as the process goes through its release and vote phases.

-Steve

Re: Looking to a Hadoop 3 release

[Steve Loughran <st...@hortonworks.com>]

> On 19 Feb 2016, at 11:27, Dmitry Sivachenko <tr...@gmail.com> wrote:
> 
> 
>> On 19 Feb 2016, at 01:35, Andrew Wang <an...@cloudera.com> wrote:
>> 
>> Hi all,
>> 
>> Reviving this thread. I've seen renewed interest in a trunk release since
>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
>> shell script rewrite, and many other improvements, I think it's time to
>> revisit Hadoop 3.0 release plans.
>> 
> 

It's time to start ... I suspect it'll take a while to stabilise. I look forward to the new shell scripts already

One thing I do want there is for all the alpha releases to make clear that there are no compatibility policies here; protocols may change and there is no requirement of the first 3.x release to be compatible with all the 3.0.x alphas. That's something we missed out on the 2.0.x-alpha process, or at least not repeated often enough.

> 
> Hello,
> 
> any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes out?
> 
> Thanks!
> 
> 

sounds like a good time for a status update on the FB work —and anything people can do to test it would be appreciated by all. That includes testing on ipv4 systems, and especially, IPv4/v6 systems with Kerberos turned on and both MIT and AD kerberos servers. At the same time, IPv6 support ought to be something that could be added in.


I don't have any opinions on timescale, but

+1 to anything related to classpath isolation
+1 to a careful bump of versions of dependencies.
+1 to fixing the outstanding Java 8 migration issues, especially the big Jersey patch that's just been updated.
+1 to switching to JIRA-created release notes

Having been doing the slider releases recently, it's clear to me that you can do a lot in automating the release process itself. All those steps in the release runbook can be turned into targets in a special ant release.xml build file, calling maven, gpg, etc.

I think doing something like this for 3.0 will significantly benefit both the release phase here but the future releases

This is the slider one: https://github.com/apache/incubator-slider/blob/develop/bin/release.xml

It doesn't replace maven, instead it choreographs that along with all the other steps: signing and checksumming artifacts, publishing them, voting

it includes
 -refusing to release if the git repo is modified
 -making the various git branch/tag/push operations
 -issuing the various mvn versions:update commands
 -signing
 -publishing via asf SVN 
 -using GET calls too verify the artifacts made it
 -generating the vote and vote result emails (it even counts the votes)

I recommend this is included as part of the release process. It does make a difference; we can now cut new releases with no human intervention other than editing a properties file and running different targets as the process goes through its release and vote phases.

-Steve

Re: Looking to a Hadoop 3 release

[Steve Loughran <st...@hortonworks.com>]

> On 19 Feb 2016, at 11:27, Dmitry Sivachenko <tr...@gmail.com> wrote:
> 
> 
>> On 19 Feb 2016, at 01:35, Andrew Wang <an...@cloudera.com> wrote:
>> 
>> Hi all,
>> 
>> Reviving this thread. I've seen renewed interest in a trunk release since
>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
>> shell script rewrite, and many other improvements, I think it's time to
>> revisit Hadoop 3.0 release plans.
>> 
> 

It's time to start ... I suspect it'll take a while to stabilise. I look forward to the new shell scripts already

One thing I do want there is for all the alpha releases to make clear that there are no compatibility policies here; protocols may change and there is no requirement of the first 3.x release to be compatible with all the 3.0.x alphas. That's something we missed out on the 2.0.x-alpha process, or at least not repeated often enough.

> 
> Hello,
> 
> any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes out?
> 
> Thanks!
> 
> 

sounds like a good time for a status update on the FB work —and anything people can do to test it would be appreciated by all. That includes testing on ipv4 systems, and especially, IPv4/v6 systems with Kerberos turned on and both MIT and AD kerberos servers. At the same time, IPv6 support ought to be something that could be added in.


I don't have any opinions on timescale, but

+1 to anything related to classpath isolation
+1 to a careful bump of versions of dependencies.
+1 to fixing the outstanding Java 8 migration issues, especially the big Jersey patch that's just been updated.
+1 to switching to JIRA-created release notes

Having been doing the slider releases recently, it's clear to me that you can do a lot in automating the release process itself. All those steps in the release runbook can be turned into targets in a special ant release.xml build file, calling maven, gpg, etc.

I think doing something like this for 3.0 will significantly benefit both the release phase here but the future releases

This is the slider one: https://github.com/apache/incubator-slider/blob/develop/bin/release.xml

It doesn't replace maven, instead it choreographs that along with all the other steps: signing and checksumming artifacts, publishing them, voting

it includes
 -refusing to release if the git repo is modified
 -making the various git branch/tag/push operations
 -issuing the various mvn versions:update commands
 -signing
 -publishing via asf SVN 
 -using GET calls too verify the artifacts made it
 -generating the vote and vote result emails (it even counts the votes)

I recommend this is included as part of the release process. It does make a difference; we can now cut new releases with no human intervention other than editing a properties file and running different targets as the process goes through its release and vote phases.

-Steve

Re: Looking to a Hadoop 3 release

[Dmitry Sivachenko <tr...@gmail.com>]

> On 19 Feb 2016, at 01:35, Andrew Wang <an...@cloudera.com> wrote:
> 
> Hi all,
> 
> Reviving this thread. I've seen renewed interest in a trunk release since
> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
> shell script rewrite, and many other improvements, I think it's time to
> revisit Hadoop 3.0 release plans.
> 


Hello,

any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes out?

Thanks!