You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Robert Kanter <rk...@cloudera.com> on 2016/09/30 00:56:27 UTC

Oozie going forward

Hi,

Now that Oozie 4.3 is branched and will be happening soon, I imagine the
next release we do with be Oozie 5 with Oozie On Yarn.  We can continue to
keep OYA in a feature branch until later when it's more stable so we don't
disrupt the master branch in the meantime.  We've already removed 0.23 and
made Hadoop 2 the default.  Removing Hadoop 1 will simplify things, and is
currently in the way of other tasks like upgrading HBase.  I imagine it
will continue to do so more as time goes on.

I'd like to propose the following:

   1. Oozie 4.3 is the last release to support Hadoop 1.  If we want to do
   another Oozie 4.x release later, we can branch it from branch-4.3 instead
   of trunk.  Though I expect 4.3 will be the last minor release of Oozie 4.
   2. Oozie 5 (the master branch; aka trunk) will be the first release to
   support only Hadoop 2 and Hadoop 3 (these should be much more similar than
   Hadoop 1 to 2 was; I expect things will work as-is by just changing the
   Hadoop version so we probably won't even need profiles).  We'll have Oozie
   On Yarn as the marquee feature.
   3. We'll drop Hadoop 1 from the master branch.  We can reuse parts
   of OOZIE-2316.
   4. The master branch will become 5.0.0-SNAPSHOT instead of
   4.4.0-SNAPSHOT.

I'd like us to take care of #3 soon, because it's getting in the way.  We
might as well also take care of #4 soon too.

Thoughts?


thanks
- Robert

Re: Oozie going forward

Posted by Peter Bacsko <pb...@cloudera.com>.
Hi,

one question from me: how shall we integrate our OYA changes to master?

I can think of four options:
1. Simply git merge our commits from the oya branch. This will mean at
least 20-30 commits.
2. We squash our commits to have fewer and then merge the oya branch (right
now I can't say in advance how many commits we'll end up with)
3. We squash _all_ commits into one and then merge/cherry-pick it.
4. We create a patch file containing all changes and apply it

Ideas, suggestions are welcome.

Peter

On Fri, Sep 30, 2016 at 2:56 AM, Robert Kanter <rk...@cloudera.com> wrote:

> Hi,
>
> Now that Oozie 4.3 is branched and will be happening soon, I imagine the
> next release we do with be Oozie 5 with Oozie On Yarn.  We can continue to
> keep OYA in a feature branch until later when it's more stable so we don't
> disrupt the master branch in the meantime.  We've already removed 0.23 and
> made Hadoop 2 the default.  Removing Hadoop 1 will simplify things, and is
> currently in the way of other tasks like upgrading HBase.  I imagine it
> will continue to do so more as time goes on.
>
> I'd like to propose the following:
>
>    1. Oozie 4.3 is the last release to support Hadoop 1.  If we want to do
>    another Oozie 4.x release later, we can branch it from branch-4.3
> instead
>    of trunk.  Though I expect 4.3 will be the last minor release of Oozie
> 4.
>    2. Oozie 5 (the master branch; aka trunk) will be the first release to
>    support only Hadoop 2 and Hadoop 3 (these should be much more similar
> than
>    Hadoop 1 to 2 was; I expect things will work as-is by just changing the
>    Hadoop version so we probably won't even need profiles).  We'll have
> Oozie
>    On Yarn as the marquee feature.
>    3. We'll drop Hadoop 1 from the master branch.  We can reuse parts
>    of OOZIE-2316.
>    4. The master branch will become 5.0.0-SNAPSHOT instead of
>    4.4.0-SNAPSHOT.
>
> I'd like us to take care of #3 soon, because it's getting in the way.  We
> might as well also take care of #4 soon too.
>
> Thoughts?
>
>
> thanks
> - Robert
>

Re: Oozie going forward

Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Very good Rohini.+1 from me as well.
 

    On Friday, September 30, 2016 1:11 PM, Rohini Palaniswamy <ro...@gmail.com> wrote:
 

 Abhishek,  I did rename the 4.4.0 version to 5.0.0 now. Work on a feature branch will still be marked with the "oya" version. When it is merged, then the 5.0.0 version can be added in addition to the "oya" version. This is usually what is done for work in feature branches.
Peter,  It will have to be 4) We create a patch file containing all changes and apply it.   You will have to issue a review request with all the changes in the oya branch for review by the wider community before committing it to master.
Regards,Rohini

On Fri, Sep 30, 2016 at 1:00 PM, Abhishek Bafna <ba...@gmail.com> wrote:

Hi Robert,

Can you please create a fix version 5.0.0, so that OYA related stuff can be tagged correctly.

Thanks,
Abhishek
> On Oct 1, 2016, at 12:13 AM, Mohammad Islam <mi...@yahoo.com.INVALID> wrote:
>
> Robert,Very good plan!
> How other  products (pig/hive etc) are handling the Hadoop 1 issue?If they are planning in the same way, this will be easy for us.
> Regards,Mohammad
>
>
>
>
>    On Thursday, September 29, 2016 5:56 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
>
> Hi,
>
> Now that Oozie 4.3 is branched and will be happening soon, I imagine the
> next release we do with be Oozie 5 with Oozie On Yarn.  We can continue to
> keep OYA in a feature branch until later when it's more stable so we don't
> disrupt the master branch in the meantime.  We've already removed 0.23 and
> made Hadoop 2 the default.  Removing Hadoop 1 will simplify things, and is
> currently in the way of other tasks like upgrading HBase.  I imagine it
> will continue to do so more as time goes on.
>
> I'd like to propose the following:
>
>   1. Oozie 4.3 is the last release to support Hadoop 1.  If we want to do
>   another Oozie 4.x release later, we can branch it from branch-4.3 instead
>   of trunk.  Though I expect 4.3 will be the last minor release of Oozie 4.
>   2. Oozie 5 (the master branch; aka trunk) will be the first release to
>   support only Hadoop 2 and Hadoop 3 (these should be much more similar than
>   Hadoop 1 to 2 was; I expect things will work as-is by just changing the
>   Hadoop version so we probably won't even need profiles).  We'll have Oozie
>   On Yarn as the marquee feature.
>   3. We'll drop Hadoop 1 from the master branch.  We can reuse parts
>   of OOZIE-2316.
>   4. The master branch will become 5.0.0-SNAPSHOT instead of
>   4.4.0-SNAPSHOT.
>
> I'd like us to take care of #3 soon, because it's getting in the way.  We
> might as well also take care of #4 soon too.
>
> Thoughts?
>
>
> thanks
> - Robert
>
>





   

Re: Oozie going forward

Posted by Rohini Palaniswamy <ro...@gmail.com>.
Abhishek,
  I did rename the 4.4.0 version to 5.0.0 now. Work on a feature branch
will still be marked with the "oya" version. When it is merged, then the
5.0.0 version can be added in addition to the "oya" version. This is
usually what is done for work in feature branches.

Peter,
  It will have to be 4) We create a patch file containing all changes and
apply it.   You will have to issue a review request with all the changes in
the oya branch for review by the wider community before committing it to
master.

Regards,
Rohini

On Fri, Sep 30, 2016 at 1:00 PM, Abhishek Bafna <ba...@gmail.com>
wrote:

> Hi Robert,
>
> Can you please create a fix version 5.0.0, so that OYA related stuff can
> be tagged correctly.
>
> Thanks,
> Abhishek
> > On Oct 1, 2016, at 12:13 AM, Mohammad Islam <mi...@yahoo.com.INVALID>
> wrote:
> >
> > Robert,Very good plan!
> > How other  products (pig/hive etc) are handling the Hadoop 1 issue?If
> they are planning in the same way, this will be easy for us.
> > Regards,Mohammad
> >
> >
> >
> >
> >    On Thursday, September 29, 2016 5:56 PM, Robert Kanter <
> rkanter@cloudera.com> wrote:
> >
> >
> > Hi,
> >
> > Now that Oozie 4.3 is branched and will be happening soon, I imagine the
> > next release we do with be Oozie 5 with Oozie On Yarn.  We can continue
> to
> > keep OYA in a feature branch until later when it's more stable so we
> don't
> > disrupt the master branch in the meantime.  We've already removed 0.23
> and
> > made Hadoop 2 the default.  Removing Hadoop 1 will simplify things, and
> is
> > currently in the way of other tasks like upgrading HBase.  I imagine it
> > will continue to do so more as time goes on.
> >
> > I'd like to propose the following:
> >
> >   1. Oozie 4.3 is the last release to support Hadoop 1.  If we want to do
> >   another Oozie 4.x release later, we can branch it from branch-4.3
> instead
> >   of trunk.  Though I expect 4.3 will be the last minor release of Oozie
> 4.
> >   2. Oozie 5 (the master branch; aka trunk) will be the first release to
> >   support only Hadoop 2 and Hadoop 3 (these should be much more similar
> than
> >   Hadoop 1 to 2 was; I expect things will work as-is by just changing the
> >   Hadoop version so we probably won't even need profiles).  We'll have
> Oozie
> >   On Yarn as the marquee feature.
> >   3. We'll drop Hadoop 1 from the master branch.  We can reuse parts
> >   of OOZIE-2316.
> >   4. The master branch will become 5.0.0-SNAPSHOT instead of
> >   4.4.0-SNAPSHOT.
> >
> > I'd like us to take care of #3 soon, because it's getting in the way.  We
> > might as well also take care of #4 soon too.
> >
> > Thoughts?
> >
> >
> > thanks
> > - Robert
> >
> >
>
>

Re: Oozie going forward

Posted by Abhishek Bafna <ba...@gmail.com>.
Hi Robert,

Can you please create a fix version 5.0.0, so that OYA related stuff can be tagged correctly.

Thanks,
Abhishek
> On Oct 1, 2016, at 12:13 AM, Mohammad Islam <mi...@yahoo.com.INVALID> wrote:
> 
> Robert,Very good plan!
> How other  products (pig/hive etc) are handling the Hadoop 1 issue?If they are planning in the same way, this will be easy for us.
> Regards,Mohammad
> 
> 
> 
> 
>    On Thursday, September 29, 2016 5:56 PM, Robert Kanter <rk...@cloudera.com> wrote:
> 
> 
> Hi,
> 
> Now that Oozie 4.3 is branched and will be happening soon, I imagine the
> next release we do with be Oozie 5 with Oozie On Yarn.  We can continue to
> keep OYA in a feature branch until later when it's more stable so we don't
> disrupt the master branch in the meantime.  We've already removed 0.23 and
> made Hadoop 2 the default.  Removing Hadoop 1 will simplify things, and is
> currently in the way of other tasks like upgrading HBase.  I imagine it
> will continue to do so more as time goes on.
> 
> I'd like to propose the following:
> 
>   1. Oozie 4.3 is the last release to support Hadoop 1.  If we want to do
>   another Oozie 4.x release later, we can branch it from branch-4.3 instead
>   of trunk.  Though I expect 4.3 will be the last minor release of Oozie 4.
>   2. Oozie 5 (the master branch; aka trunk) will be the first release to
>   support only Hadoop 2 and Hadoop 3 (these should be much more similar than
>   Hadoop 1 to 2 was; I expect things will work as-is by just changing the
>   Hadoop version so we probably won't even need profiles).  We'll have Oozie
>   On Yarn as the marquee feature.
>   3. We'll drop Hadoop 1 from the master branch.  We can reuse parts
>   of OOZIE-2316.
>   4. The master branch will become 5.0.0-SNAPSHOT instead of
>   4.4.0-SNAPSHOT.
> 
> I'd like us to take care of #3 soon, because it's getting in the way.  We
> might as well also take care of #4 soon too.
> 
> Thoughts?
> 
> 
> thanks
> - Robert
> 
> 


Re: Oozie going forward

Posted by Rohini Palaniswamy <ro...@gmail.com>.
+1. Pig 0.16 was the last release to support Hadoop 1.x. The next release
will not support it. Hive has already dropped it from the Hive 2.x releases.

On Fri, Sep 30, 2016 at 11:43 AM, Mohammad Islam <mislam77@yahoo.com.invalid
> wrote:

> Robert,Very good plan!
> How other  products (pig/hive etc) are handling the Hadoop 1 issue?If they
> are planning in the same way, this will be easy for us.
> Regards,Mohammad
>
>
>
>
>     On Thursday, September 29, 2016 5:56 PM, Robert Kanter <
> rkanter@cloudera.com> wrote:
>
>
>  Hi,
>
> Now that Oozie 4.3 is branched and will be happening soon, I imagine the
> next release we do with be Oozie 5 with Oozie On Yarn.  We can continue to
> keep OYA in a feature branch until later when it's more stable so we don't
> disrupt the master branch in the meantime.  We've already removed 0.23 and
> made Hadoop 2 the default.  Removing Hadoop 1 will simplify things, and is
> currently in the way of other tasks like upgrading HBase.  I imagine it
> will continue to do so more as time goes on.
>
> I'd like to propose the following:
>
>   1. Oozie 4.3 is the last release to support Hadoop 1.  If we want to do
>   another Oozie 4.x release later, we can branch it from branch-4.3 instead
>   of trunk.  Though I expect 4.3 will be the last minor release of Oozie 4.
>   2. Oozie 5 (the master branch; aka trunk) will be the first release to
>   support only Hadoop 2 and Hadoop 3 (these should be much more similar
> than
>   Hadoop 1 to 2 was; I expect things will work as-is by just changing the
>   Hadoop version so we probably won't even need profiles).  We'll have
> Oozie
>   On Yarn as the marquee feature.
>   3. We'll drop Hadoop 1 from the master branch.  We can reuse parts
>   of OOZIE-2316.
>   4. The master branch will become 5.0.0-SNAPSHOT instead of
>   4.4.0-SNAPSHOT.
>
> I'd like us to take care of #3 soon, because it's getting in the way.  We
> might as well also take care of #4 soon too.
>
> Thoughts?
>
>
> thanks
> - Robert
>
>
>
>

Re: Oozie going forward

Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Robert,Very good plan!
How other  products (pig/hive etc) are handling the Hadoop 1 issue?If they are planning in the same way, this will be easy for us.
Regards,Mohammad


 

    On Thursday, September 29, 2016 5:56 PM, Robert Kanter <rk...@cloudera.com> wrote:
 

 Hi,

Now that Oozie 4.3 is branched and will be happening soon, I imagine the
next release we do with be Oozie 5 with Oozie On Yarn.  We can continue to
keep OYA in a feature branch until later when it's more stable so we don't
disrupt the master branch in the meantime.  We've already removed 0.23 and
made Hadoop 2 the default.  Removing Hadoop 1 will simplify things, and is
currently in the way of other tasks like upgrading HBase.  I imagine it
will continue to do so more as time goes on.

I'd like to propose the following:

  1. Oozie 4.3 is the last release to support Hadoop 1.  If we want to do
  another Oozie 4.x release later, we can branch it from branch-4.3 instead
  of trunk.  Though I expect 4.3 will be the last minor release of Oozie 4.
  2. Oozie 5 (the master branch; aka trunk) will be the first release to
  support only Hadoop 2 and Hadoop 3 (these should be much more similar than
  Hadoop 1 to 2 was; I expect things will work as-is by just changing the
  Hadoop version so we probably won't even need profiles).  We'll have Oozie
  On Yarn as the marquee feature.
  3. We'll drop Hadoop 1 from the master branch.  We can reuse parts
  of OOZIE-2316.
  4. The master branch will become 5.0.0-SNAPSHOT instead of
  4.4.0-SNAPSHOT.

I'd like us to take care of #3 soon, because it's getting in the way.  We
might as well also take care of #4 soon too.

Thoughts?


thanks
- Robert