Slider and Hadoop versions

[Steve Loughran <st...@hortonworks.com>]

A quick update on Hadoop versions

Slider 0.40: works on Hadoop 2.4.0+; but AM restart doesn't work properly
on Hadoop 2.4

Hadoop 2.5 is about to RC; I want to do a couple of tests there as part of
the Hadoop release process -Slider is the defacto AM failure/restart
functional test, so we get to verify it works.

Hadoop 2.5 is also in sync with artifacts  we need. (especially jackson,
https://issues.apache.org/jira/browse/HADOOP-10104 ) -this syncs it up with
Curator. FWIW that update broke Tez as they were uploading the jackson JARs
themselves ). Tez worked because their Jars were the Hadoop 2.4 versions,
and now they are unhappy about the change (
https://issues.apache.org/jira/browse/YARN-2092 ). My stance is that we
can't lock down Hadoop dependencies for all YARn apps, so unless we do
better isolation/dcache all of us YARN-app authors who want to use the
classpath of the target machine will have to deal with that.

Hadoop 2.6 is the next Hadoop release after. This has some things we'd like
 -Credential Provider: https://issues.apache.org/jira/browse/HADOOP-10607
 -hopefully, YARN service registry (
https://issues.apache.org/jira/browse/YARN-913). I have a proto registry
design/imple there which I want to stick up for comments by the end of the
week. There'a a branch in features/ which uses this...I'm using slider as a
way of seeing if the API is usable
 -Java 7 (once I push in that patch)

Once Hadoop 2.5 is out I plan to switch to Hadoop 2.6-SNAPSHOT as the base
version for slider dev.. good for Slider and Hadoop itself (we test the
features), but it would mean that releases after that point won't even work
on Hadoop 2.5

I'm going to propose, therefore:

1. We cut a Slider 0.50 release to immediately follow the Hadoop 2.5
release, as the one-and-only Hadoop 2.5+ version; Slider 0.40 will be the
last Hadoop 2.4 release. There's no actual source difference between them,
we'd just build, test and release against the 2.5 JARs. I already test
against branch-2 on Java 8 on one VM, so don't expect surprises.

2. After that release, we switch develop/ to build against Hadoop branch-2
only. Either developers build locally, or they do builds against the apache
snapshot repository

Thoughts?

-steve

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Slider and Hadoop versions

[Billie Rinaldi <bi...@gmail.com>]

Sounds good to me.  There is a similar issue with our Accumulo dependency,
which is that Accumulo 1.6.1-SNAPSHOT takes advantage of
CredentialProvider.  I'd like to switch to 1.6.1-SNAPSHOT when we switch
Hadoop to 2.6.0-SNAPSHOT.  If Accumulo hasn't been released by the time
Hadoop 2.6.0 is out, I'll consider rolling back the changes to be
compatible with 1.6.0.


On Tue, Aug 5, 2014 at 2:31 AM, Steve Loughran <st...@hortonworks.com>
wrote:

> A quick update on Hadoop versions
>
> Slider 0.40: works on Hadoop 2.4.0+; but AM restart doesn't work properly
> on Hadoop 2.4
>
> Hadoop 2.5 is about to RC; I want to do a couple of tests there as part of
> the Hadoop release process -Slider is the defacto AM failure/restart
> functional test, so we get to verify it works.
>
> Hadoop 2.5 is also in sync with artifacts  we need. (especially jackson,
> https://issues.apache.org/jira/browse/HADOOP-10104 ) -this syncs it up
> with
> Curator. FWIW that update broke Tez as they were uploading the jackson JARs
> themselves ). Tez worked because their Jars were the Hadoop 2.4 versions,
> and now they are unhappy about the change (
> https://issues.apache.org/jira/browse/YARN-2092 ). My stance is that we
> can't lock down Hadoop dependencies for all YARn apps, so unless we do
> better isolation/dcache all of us YARN-app authors who want to use the
> classpath of the target machine will have to deal with that.
>
> Hadoop 2.6 is the next Hadoop release after. This has some things we'd like
>  -Credential Provider: https://issues.apache.org/jira/browse/HADOOP-10607
>  -hopefully, YARN service registry (
> https://issues.apache.org/jira/browse/YARN-913). I have a proto registry
> design/imple there which I want to stick up for comments by the end of the
> week. There'a a branch in features/ which uses this...I'm using slider as a
> way of seeing if the API is usable
>  -Java 7 (once I push in that patch)
>
> Once Hadoop 2.5 is out I plan to switch to Hadoop 2.6-SNAPSHOT as the base
> version for slider dev.. good for Slider and Hadoop itself (we test the
> features), but it would mean that releases after that point won't even work
> on Hadoop 2.5
>
> I'm going to propose, therefore:
>
> 1. We cut a Slider 0.50 release to immediately follow the Hadoop 2.5
> release, as the one-and-only Hadoop 2.5+ version; Slider 0.40 will be the
> last Hadoop 2.4 release. There's no actual source difference between them,
> we'd just build, test and release against the 2.5 JARs. I already test
> against branch-2 on Java 8 on one VM, so don't expect surprises.
>
> 2. After that release, we switch develop/ to build against Hadoop branch-2
> only. Either developers build locally, or they do builds against the apache
> snapshot repository
>
> Thoughts?
>
> -steve
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Slider and Hadoop versions

[Gour Saha <gs...@hortonworks.com>]

I had to slap Hadoop-yarn 2.4.1 libraries on top of slider 0.40 to get the
AM restart feature working. It's kind of a hack, so +1 for the plan to move
slider to hadoop 2.5.

-Gour


On Tue, Aug 5, 2014 at 6:41 AM, Jon Maron <jm...@hortonworks.com> wrote:

> Sounds liike a viable plan.  Do we know the hadoop version that is planned
> for Champlain?
>
> — Jon
>
> On Aug 5, 2014, at 5:31 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> > A quick update on Hadoop versions
> >
> > Slider 0.40: works on Hadoop 2.4.0+; but AM restart doesn't work properly
> > on Hadoop 2.4
> >
> > Hadoop 2.5 is about to RC; I want to do a couple of tests there as part
> of
> > the Hadoop release process -Slider is the defacto AM failure/restart
> > functional test, so we get to verify it works.
> >
> > Hadoop 2.5 is also in sync with artifacts  we need. (especially jackson,
> > https://issues.apache.org/jira/browse/HADOOP-10104 ) -this syncs it up
> with
> > Curator. FWIW that update broke Tez as they were uploading the jackson
> JARs
> > themselves ). Tez worked because their Jars were the Hadoop 2.4 versions,
> > and now they are unhappy about the change (
> > https://issues.apache.org/jira/browse/YARN-2092 ). My stance is that we
> > can't lock down Hadoop dependencies for all YARn apps, so unless we do
> > better isolation/dcache all of us YARN-app authors who want to use the
> > classpath of the target machine will have to deal with that.
> >
> > Hadoop 2.6 is the next Hadoop release after. This has some things we'd
> like
> > -Credential Provider: https://issues.apache.org/jira/browse/HADOOP-10607
> > -hopefully, YARN service registry (
> > https://issues.apache.org/jira/browse/YARN-913). I have a proto registry
> > design/imple there which I want to stick up for comments by the end of
> the
> > week. There'a a branch in features/ which uses this...I'm using slider
> as a
> > way of seeing if the API is usable
> > -Java 7 (once I push in that patch)
> >
> > Once Hadoop 2.5 is out I plan to switch to Hadoop 2.6-SNAPSHOT as the
> base
> > version for slider dev.. good for Slider and Hadoop itself (we test the
> > features), but it would mean that releases after that point won't even
> work
> > on Hadoop 2.5
> >
> > I'm going to propose, therefore:
> >
> > 1. We cut a Slider 0.50 release to immediately follow the Hadoop 2.5
> > release, as the one-and-only Hadoop 2.5+ version; Slider 0.40 will be the
> > last Hadoop 2.4 release. There's no actual source difference between
> them,
> > we'd just build, test and release against the 2.5 JARs. I already test
> > against branch-2 on Java 8 on one VM, so don't expect surprises.
> >
> > 2. After that release, we switch develop/ to build against Hadoop
> branch-2
> > only. Either developers build locally, or they do builds against the
> apache
> > snapshot repository
> >
> > Thoughts?
> >
> > -steve
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Slider and Hadoop versions

[Jon Maron <jm...@hortonworks.com>]

Sounds liike a viable plan.  Do we know the hadoop version that is planned for Champlain?

— Jon

On Aug 5, 2014, at 5:31 AM, Steve Loughran <st...@hortonworks.com> wrote:

> A quick update on Hadoop versions
> 
> Slider 0.40: works on Hadoop 2.4.0+; but AM restart doesn't work properly
> on Hadoop 2.4
> 
> Hadoop 2.5 is about to RC; I want to do a couple of tests there as part of
> the Hadoop release process -Slider is the defacto AM failure/restart
> functional test, so we get to verify it works.
> 
> Hadoop 2.5 is also in sync with artifacts  we need. (especially jackson,
> https://issues.apache.org/jira/browse/HADOOP-10104 ) -this syncs it up with
> Curator. FWIW that update broke Tez as they were uploading the jackson JARs
> themselves ). Tez worked because their Jars were the Hadoop 2.4 versions,
> and now they are unhappy about the change (
> https://issues.apache.org/jira/browse/YARN-2092 ). My stance is that we
> can't lock down Hadoop dependencies for all YARn apps, so unless we do
> better isolation/dcache all of us YARN-app authors who want to use the
> classpath of the target machine will have to deal with that.
> 
> Hadoop 2.6 is the next Hadoop release after. This has some things we'd like
> -Credential Provider: https://issues.apache.org/jira/browse/HADOOP-10607
> -hopefully, YARN service registry (
> https://issues.apache.org/jira/browse/YARN-913). I have a proto registry
> design/imple there which I want to stick up for comments by the end of the
> week. There'a a branch in features/ which uses this...I'm using slider as a
> way of seeing if the API is usable
> -Java 7 (once I push in that patch)
> 
> Once Hadoop 2.5 is out I plan to switch to Hadoop 2.6-SNAPSHOT as the base
> version for slider dev.. good for Slider and Hadoop itself (we test the
> features), but it would mean that releases after that point won't even work
> on Hadoop 2.5
> 
> I'm going to propose, therefore:
> 
> 1. We cut a Slider 0.50 release to immediately follow the Hadoop 2.5
> release, as the one-and-only Hadoop 2.5+ version; Slider 0.40 will be the
> last Hadoop 2.4 release. There's no actual source difference between them,
> we'd just build, test and release against the 2.5 JARs. I already test
> against branch-2 on Java 8 on one VM, so don't expect surprises.
> 
> 2. After that release, we switch develop/ to build against Hadoop branch-2
> only. Either developers build locally, or they do builds against the apache
> snapshot repository
> 
> Thoughts?
> 
> -steve
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.