Hadoop 3, Oozie 5

[Peter Cseh <ge...@cloudera.com>]

Hi everyone,

As we continue the development of Oozie 5 we would like to ensure it works
with Hadoop 3.

To achieve this, I think it would be good to improve our pre-commit job to
test that Oozie master compiles with the newer (alpha/beta or even
snapshot) versions of Hadoop, Hive, Pig, HBase and Spark.
Also, I would like to have an Oozie 5 beta1 release which comes out in sync
with Hadoop 3 beta1
<https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release>,
currently planned at the beginning of July.

What do you guys think about Hadoop 3 integration?
gp



-- 
Peter Cseh
Software Engineer
<http://www.cloudera.com>

Re: Hadoop 3, Oozie 5

[Robert Kanter <rk...@cloudera.com>]

+1 for updating the pre-commit job.  Hadoop 2 and 3 are much more similar
than Hadoop 1 and 2 were, so we should have less complicated pom stuff -
probably just the Hadoop versions and maybe a few other versions.

We could also look into using Yetus <https://yetus.apache.org/>, which is
what Hadoop uses for pre-commit.  Hadoop lets you run a patch against a
different branch based on the name, so it certainly has this capability.
It might be good to invest time into using it instead of continuing to
maintain our own scripts.

Doing an Oozie 5 beta 1 release at the time of Hadoop 3 beta 1 sounds like
a good idea.  Depending on how quickly OYA development continues, it may be
a bit optimistic, but we can still shoot for it and evaluate closer to the
time.

- Robert

On Fri, May 26, 2017 at 7:20 AM, Artem Ervits <ar...@gmail.com> wrote:

> +1, for a wider ecosystem integration testing, should this effort be part
> of Apache Bigtop?
>
> On Fri, May 26, 2017 at 8:57 AM, Peter Cseh <ge...@cloudera.com> wrote:
>
> > Hi everyone,
> >
> > As we continue the development of Oozie 5 we would like to ensure it
> works
> > with Hadoop 3.
> >
> > To achieve this, I think it would be good to improve our pre-commit job
> to
> > test that Oozie master compiles with the newer (alpha/beta or even
> > snapshot) versions of Hadoop, Hive, Pig, HBase and Spark.
> > Also, I would like to have an Oozie 5 beta1 release which comes out in
> sync
> > with Hadoop 3 beta1
> > <https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release
> >,
> > currently planned at the beginning of July.
> >
> > What do you guys think about Hadoop 3 integration?
> > gp
> >
> >
> >
> > --
> > Peter Cseh
> > Software Engineer
> > <http://www.cloudera.com>
> >
>

Re: Hadoop 3, Oozie 5

[Artem Ervits <ar...@gmail.com>]

+1, for a wider ecosystem integration testing, should this effort be part
of Apache Bigtop?

On Fri, May 26, 2017 at 8:57 AM, Peter Cseh <ge...@cloudera.com> wrote:

> Hi everyone,
>
> As we continue the development of Oozie 5 we would like to ensure it works
> with Hadoop 3.
>
> To achieve this, I think it would be good to improve our pre-commit job to
> test that Oozie master compiles with the newer (alpha/beta or even
> snapshot) versions of Hadoop, Hive, Pig, HBase and Spark.
> Also, I would like to have an Oozie 5 beta1 release which comes out in sync
> with Hadoop 3 beta1
> <https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release>,
> currently planned at the beginning of July.
>
> What do you guys think about Hadoop 3 integration?
> gp
>
>
>
> --
> Peter Cseh
> Software Engineer
> <http://www.cloudera.com>
>