You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Robert Kanter <rk...@cloudera.com> on 2016/09/30 00:56:27 UTC
Oozie going forward
Hi,
Now that Oozie 4.3 is branched and will be happening soon, I imagine the
next release we do with be Oozie 5 with Oozie On Yarn. We can continue to
keep OYA in a feature branch until later when it's more stable so we don't
disrupt the master branch in the meantime. We've already removed 0.23 and
made Hadoop 2 the default. Removing Hadoop 1 will simplify things, and is
currently in the way of other tasks like upgrading HBase. I imagine it
will continue to do so more as time goes on.
I'd like to propose the following:
1. Oozie 4.3 is the last release to support Hadoop 1. If we want to do
another Oozie 4.x release later, we can branch it from branch-4.3 instead
of trunk. Though I expect 4.3 will be the last minor release of Oozie 4.
2. Oozie 5 (the master branch; aka trunk) will be the first release to
support only Hadoop 2 and Hadoop 3 (these should be much more similar than
Hadoop 1 to 2 was; I expect things will work as-is by just changing the
Hadoop version so we probably won't even need profiles). We'll have Oozie
On Yarn as the marquee feature.
3. We'll drop Hadoop 1 from the master branch. We can reuse parts
of OOZIE-2316.
4. The master branch will become 5.0.0-SNAPSHOT instead of
4.4.0-SNAPSHOT.
I'd like us to take care of #3 soon, because it's getting in the way. We
might as well also take care of #4 soon too.
Thoughts?
thanks
- Robert
Re: Oozie going forward
Posted by Peter Bacsko <pb...@cloudera.com>.
Hi,
one question from me: how shall we integrate our OYA changes to master?
I can think of four options:
1. Simply git merge our commits from the oya branch. This will mean at
least 20-30 commits.
2. We squash our commits to have fewer and then merge the oya branch (right
now I can't say in advance how many commits we'll end up with)
3. We squash _all_ commits into one and then merge/cherry-pick it.
4. We create a patch file containing all changes and apply it
Ideas, suggestions are welcome.
Peter
On Fri, Sep 30, 2016 at 2:56 AM, Robert Kanter <rk...@cloudera.com> wrote:
> Hi,
>
> Now that Oozie 4.3 is branched and will be happening soon, I imagine the
> next release we do with be Oozie 5 with Oozie On Yarn. We can continue to
> keep OYA in a feature branch until later when it's more stable so we don't
> disrupt the master branch in the meantime. We've already removed 0.23 and
> made Hadoop 2 the default. Removing Hadoop 1 will simplify things, and is
> currently in the way of other tasks like upgrading HBase. I imagine it
> will continue to do so more as time goes on.
>
> I'd like to propose the following:
>
> 1. Oozie 4.3 is the last release to support Hadoop 1. If we want to do
> another Oozie 4.x release later, we can branch it from branch-4.3
> instead
> of trunk. Though I expect 4.3 will be the last minor release of Oozie
> 4.
> 2. Oozie 5 (the master branch; aka trunk) will be the first release to
> support only Hadoop 2 and Hadoop 3 (these should be much more similar
> than
> Hadoop 1 to 2 was; I expect things will work as-is by just changing the
> Hadoop version so we probably won't even need profiles). We'll have
> Oozie
> On Yarn as the marquee feature.
> 3. We'll drop Hadoop 1 from the master branch. We can reuse parts
> of OOZIE-2316.
> 4. The master branch will become 5.0.0-SNAPSHOT instead of
> 4.4.0-SNAPSHOT.
>
> I'd like us to take care of #3 soon, because it's getting in the way. We
> might as well also take care of #4 soon too.
>
> Thoughts?
>
>
> thanks
> - Robert
>
Re: Oozie going forward
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Very good Rohini.+1 from me as well.
On Friday, September 30, 2016 1:11 PM, Rohini Palaniswamy <ro...@gmail.com> wrote:
Abhishek, I did rename the 4.4.0 version to 5.0.0 now. Work on a feature branch will still be marked with the "oya" version. When it is merged, then the 5.0.0 version can be added in addition to the "oya" version. This is usually what is done for work in feature branches.
Peter, It will have to be 4) We create a patch file containing all changes and apply it. You will have to issue a review request with all the changes in the oya branch for review by the wider community before committing it to master.
Regards,Rohini
On Fri, Sep 30, 2016 at 1:00 PM, Abhishek Bafna <ba...@gmail.com> wrote:
Hi Robert,
Can you please create a fix version 5.0.0, so that OYA related stuff can be tagged correctly.
Thanks,
Abhishek
> On Oct 1, 2016, at 12:13 AM, Mohammad Islam <mi...@yahoo.com.INVALID> wrote:
>
> Robert,Very good plan!
> How other products (pig/hive etc) are handling the Hadoop 1 issue?If they are planning in the same way, this will be easy for us.
> Regards,Mohammad
>
>
>
>
> On Thursday, September 29, 2016 5:56 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
>
> Hi,
>
> Now that Oozie 4.3 is branched and will be happening soon, I imagine the
> next release we do with be Oozie 5 with Oozie On Yarn. We can continue to
> keep OYA in a feature branch until later when it's more stable so we don't
> disrupt the master branch in the meantime. We've already removed 0.23 and
> made Hadoop 2 the default. Removing Hadoop 1 will simplify things, and is
> currently in the way of other tasks like upgrading HBase. I imagine it
> will continue to do so more as time goes on.
>
> I'd like to propose the following:
>
> 1. Oozie 4.3 is the last release to support Hadoop 1. If we want to do
> another Oozie 4.x release later, we can branch it from branch-4.3 instead
> of trunk. Though I expect 4.3 will be the last minor release of Oozie 4.
> 2. Oozie 5 (the master branch; aka trunk) will be the first release to
> support only Hadoop 2 and Hadoop 3 (these should be much more similar than
> Hadoop 1 to 2 was; I expect things will work as-is by just changing the
> Hadoop version so we probably won't even need profiles). We'll have Oozie
> On Yarn as the marquee feature.
> 3. We'll drop Hadoop 1 from the master branch. We can reuse parts
> of OOZIE-2316.
> 4. The master branch will become 5.0.0-SNAPSHOT instead of
> 4.4.0-SNAPSHOT.
>
> I'd like us to take care of #3 soon, because it's getting in the way. We
> might as well also take care of #4 soon too.
>
> Thoughts?
>
>
> thanks
> - Robert
>
>
Re: Oozie going forward
Posted by Rohini Palaniswamy <ro...@gmail.com>.
Abhishek,
I did rename the 4.4.0 version to 5.0.0 now. Work on a feature branch
will still be marked with the "oya" version. When it is merged, then the
5.0.0 version can be added in addition to the "oya" version. This is
usually what is done for work in feature branches.
Peter,
It will have to be 4) We create a patch file containing all changes and
apply it. You will have to issue a review request with all the changes in
the oya branch for review by the wider community before committing it to
master.
Regards,
Rohini
On Fri, Sep 30, 2016 at 1:00 PM, Abhishek Bafna <ba...@gmail.com>
wrote:
> Hi Robert,
>
> Can you please create a fix version 5.0.0, so that OYA related stuff can
> be tagged correctly.
>
> Thanks,
> Abhishek
> > On Oct 1, 2016, at 12:13 AM, Mohammad Islam <mi...@yahoo.com.INVALID>
> wrote:
> >
> > Robert,Very good plan!
> > How other products (pig/hive etc) are handling the Hadoop 1 issue?If
> they are planning in the same way, this will be easy for us.
> > Regards,Mohammad
> >
> >
> >
> >
> > On Thursday, September 29, 2016 5:56 PM, Robert Kanter <
> rkanter@cloudera.com> wrote:
> >
> >
> > Hi,
> >
> > Now that Oozie 4.3 is branched and will be happening soon, I imagine the
> > next release we do with be Oozie 5 with Oozie On Yarn. We can continue
> to
> > keep OYA in a feature branch until later when it's more stable so we
> don't
> > disrupt the master branch in the meantime. We've already removed 0.23
> and
> > made Hadoop 2 the default. Removing Hadoop 1 will simplify things, and
> is
> > currently in the way of other tasks like upgrading HBase. I imagine it
> > will continue to do so more as time goes on.
> >
> > I'd like to propose the following:
> >
> > 1. Oozie 4.3 is the last release to support Hadoop 1. If we want to do
> > another Oozie 4.x release later, we can branch it from branch-4.3
> instead
> > of trunk. Though I expect 4.3 will be the last minor release of Oozie
> 4.
> > 2. Oozie 5 (the master branch; aka trunk) will be the first release to
> > support only Hadoop 2 and Hadoop 3 (these should be much more similar
> than
> > Hadoop 1 to 2 was; I expect things will work as-is by just changing the
> > Hadoop version so we probably won't even need profiles). We'll have
> Oozie
> > On Yarn as the marquee feature.
> > 3. We'll drop Hadoop 1 from the master branch. We can reuse parts
> > of OOZIE-2316.
> > 4. The master branch will become 5.0.0-SNAPSHOT instead of
> > 4.4.0-SNAPSHOT.
> >
> > I'd like us to take care of #3 soon, because it's getting in the way. We
> > might as well also take care of #4 soon too.
> >
> > Thoughts?
> >
> >
> > thanks
> > - Robert
> >
> >
>
>
Re: Oozie going forward
Posted by Abhishek Bafna <ba...@gmail.com>.
Hi Robert,
Can you please create a fix version 5.0.0, so that OYA related stuff can be tagged correctly.
Thanks,
Abhishek
> On Oct 1, 2016, at 12:13 AM, Mohammad Islam <mi...@yahoo.com.INVALID> wrote:
>
> Robert,Very good plan!
> How other products (pig/hive etc) are handling the Hadoop 1 issue?If they are planning in the same way, this will be easy for us.
> Regards,Mohammad
>
>
>
>
> On Thursday, September 29, 2016 5:56 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
>
> Hi,
>
> Now that Oozie 4.3 is branched and will be happening soon, I imagine the
> next release we do with be Oozie 5 with Oozie On Yarn. We can continue to
> keep OYA in a feature branch until later when it's more stable so we don't
> disrupt the master branch in the meantime. We've already removed 0.23 and
> made Hadoop 2 the default. Removing Hadoop 1 will simplify things, and is
> currently in the way of other tasks like upgrading HBase. I imagine it
> will continue to do so more as time goes on.
>
> I'd like to propose the following:
>
> 1. Oozie 4.3 is the last release to support Hadoop 1. If we want to do
> another Oozie 4.x release later, we can branch it from branch-4.3 instead
> of trunk. Though I expect 4.3 will be the last minor release of Oozie 4.
> 2. Oozie 5 (the master branch; aka trunk) will be the first release to
> support only Hadoop 2 and Hadoop 3 (these should be much more similar than
> Hadoop 1 to 2 was; I expect things will work as-is by just changing the
> Hadoop version so we probably won't even need profiles). We'll have Oozie
> On Yarn as the marquee feature.
> 3. We'll drop Hadoop 1 from the master branch. We can reuse parts
> of OOZIE-2316.
> 4. The master branch will become 5.0.0-SNAPSHOT instead of
> 4.4.0-SNAPSHOT.
>
> I'd like us to take care of #3 soon, because it's getting in the way. We
> might as well also take care of #4 soon too.
>
> Thoughts?
>
>
> thanks
> - Robert
>
>
Re: Oozie going forward
Posted by Rohini Palaniswamy <ro...@gmail.com>.
+1. Pig 0.16 was the last release to support Hadoop 1.x. The next release
will not support it. Hive has already dropped it from the Hive 2.x releases.
On Fri, Sep 30, 2016 at 11:43 AM, Mohammad Islam <mislam77@yahoo.com.invalid
> wrote:
> Robert,Very good plan!
> How other products (pig/hive etc) are handling the Hadoop 1 issue?If they
> are planning in the same way, this will be easy for us.
> Regards,Mohammad
>
>
>
>
> On Thursday, September 29, 2016 5:56 PM, Robert Kanter <
> rkanter@cloudera.com> wrote:
>
>
> Hi,
>
> Now that Oozie 4.3 is branched and will be happening soon, I imagine the
> next release we do with be Oozie 5 with Oozie On Yarn. We can continue to
> keep OYA in a feature branch until later when it's more stable so we don't
> disrupt the master branch in the meantime. We've already removed 0.23 and
> made Hadoop 2 the default. Removing Hadoop 1 will simplify things, and is
> currently in the way of other tasks like upgrading HBase. I imagine it
> will continue to do so more as time goes on.
>
> I'd like to propose the following:
>
> 1. Oozie 4.3 is the last release to support Hadoop 1. If we want to do
> another Oozie 4.x release later, we can branch it from branch-4.3 instead
> of trunk. Though I expect 4.3 will be the last minor release of Oozie 4.
> 2. Oozie 5 (the master branch; aka trunk) will be the first release to
> support only Hadoop 2 and Hadoop 3 (these should be much more similar
> than
> Hadoop 1 to 2 was; I expect things will work as-is by just changing the
> Hadoop version so we probably won't even need profiles). We'll have
> Oozie
> On Yarn as the marquee feature.
> 3. We'll drop Hadoop 1 from the master branch. We can reuse parts
> of OOZIE-2316.
> 4. The master branch will become 5.0.0-SNAPSHOT instead of
> 4.4.0-SNAPSHOT.
>
> I'd like us to take care of #3 soon, because it's getting in the way. We
> might as well also take care of #4 soon too.
>
> Thoughts?
>
>
> thanks
> - Robert
>
>
>
>
Re: Oozie going forward
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Robert,Very good plan!
How other products (pig/hive etc) are handling the Hadoop 1 issue?If they are planning in the same way, this will be easy for us.
Regards,Mohammad
On Thursday, September 29, 2016 5:56 PM, Robert Kanter <rk...@cloudera.com> wrote:
Hi,
Now that Oozie 4.3 is branched and will be happening soon, I imagine the
next release we do with be Oozie 5 with Oozie On Yarn. We can continue to
keep OYA in a feature branch until later when it's more stable so we don't
disrupt the master branch in the meantime. We've already removed 0.23 and
made Hadoop 2 the default. Removing Hadoop 1 will simplify things, and is
currently in the way of other tasks like upgrading HBase. I imagine it
will continue to do so more as time goes on.
I'd like to propose the following:
1. Oozie 4.3 is the last release to support Hadoop 1. If we want to do
another Oozie 4.x release later, we can branch it from branch-4.3 instead
of trunk. Though I expect 4.3 will be the last minor release of Oozie 4.
2. Oozie 5 (the master branch; aka trunk) will be the first release to
support only Hadoop 2 and Hadoop 3 (these should be much more similar than
Hadoop 1 to 2 was; I expect things will work as-is by just changing the
Hadoop version so we probably won't even need profiles). We'll have Oozie
On Yarn as the marquee feature.
3. We'll drop Hadoop 1 from the master branch. We can reuse parts
of OOZIE-2316.
4. The master branch will become 5.0.0-SNAPSHOT instead of
4.4.0-SNAPSHOT.
I'd like us to take care of #3 soon, because it's getting in the way. We
might as well also take care of #4 soon too.
Thoughts?
thanks
- Robert