You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Alejandro Abdelnur <tu...@cloudera.com> on 2012/02/01 21:18:21 UTC
Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs into
the build
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/
-----------------------------------------------------------
Review request for oozie.
Summary
-------
The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
This addresses bug OOZIE-667.
https://issues.apache.org/jira/browse/OOZIE-667
Diffs
-----
trunk/core/pom.xml 1239282
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282
trunk/docs/src/site/twiki/AG_Install.twiki 1239282
trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282
trunk/examples/pom.xml 1239282
trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/pom.xml PRE-CREATION
trunk/pom.xml 1239282
trunk/sharelib/pom.xml 1239282
trunk/sharelib/streaming/pom.xml 1239282
trunk/src/main/assemblies/distro.xml 1239282
trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
trunk/src/main/assemblies/partial-sharelib.xml 1239282
trunk/webapp/pom.xml 1239282
Diff: https://reviews.apache.org/r/3726/diff
Testing
-------
Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
Thanks,
Alejandro
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
> On 2012-02-03 03:11:48, Mohammad Islam wrote:
> > trunk/docs/src/site/twiki/DG_QuickStart.twiki, line 39
> > <https://reviews.apache.org/r/3726/diff/3/?file=72007#file72007line39>
> >
> > is the statement correct for our discussed option #2?
forgot to update the docs. will fix.
> On 2012-02-03 03:11:48, Mohammad Islam wrote:
> > trunk/core/pom.xml, line 45
> > <https://reviews.apache.org/r/3726/diff/3/?file=72004#file72004line45>
> >
> > is it compile because oozie is building that?
this should be 'provided' i was playing around in my IDE. will fix
> On 2012-02-03 03:11:48, Mohammad Islam wrote:
> > trunk/docs/src/site/twiki/DG_QuickStart.twiki, line 51
> > <https://reviews.apache.org/r/3726/diff/3/?file=72007#file72007line51>
> >
> > Same. Is it valid for option # 2.
forgot to update the docs. will fix.
> On 2012-02-03 03:11:48, Mohammad Islam wrote:
> > trunk/docs/src/site/twiki/DG_QuickStart.twiki, line 58
> > <https://reviews.apache.org/r/3726/diff/3/?file=72007#file72007line58>
> >
> > So setup script will change too?
forgot to update the docs. will fix.
> On 2012-02-03 03:11:48, Mohammad Islam wrote:
> > trunk/src/main/assemblies/hadooplib.xml, line 32
> > <https://reviews.apache.org/r/3726/diff/3/?file=72019#file72019line32>
> >
> > about the naming: hadooplibs. should we add "oozie" before hand.
> >
having 2 TAR, oozie proper TAR and hadooplib TAR, if you expand both in the same location, you'll get hadooplibs/ in the oozie-VERSION directory.
> On 2012-02-03 03:11:48, Mohammad Islam wrote:
> > trunk/webapp/pom.xml, line 69
> > <https://reviews.apache.org/r/3726/diff/3/?file=72022#file72022line69>
> >
> > why it is changed?
because it was a wrong groupId, found this out while tuning the exclusions for the hadooplibs.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4786
-----------------------------------------------------------
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3726/
> -----------------------------------------------------------
>
> (Updated 2012-02-02 22:37:20)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
>
> mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
>
> New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
>
> Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
>
> Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
>
>
> This addresses bug OOZIE-667.
> https://issues.apache.org/jira/browse/OOZIE-667
>
>
> Diffs
> -----
>
> trunk/core/pom.xml 1239887
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
> trunk/docs/src/site/twiki/AG_Install.twiki 1239887
> trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
> trunk/examples/pom.xml 1239887
> trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/pom.xml PRE-CREATION
> trunk/pom.xml 1239887
> trunk/sharelib/pom.xml 1239887
> trunk/sharelib/streaming/pom.xml 1239887
> trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
> trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
> trunk/src/main/assemblies/partial-sharelib.xml 1239887
> trunk/webapp/pom.xml 1239887
>
> Diff: https://reviews.apache.org/r/3726/diff
>
>
> Testing
> -------
>
> Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
> On 2012-02-03 03:11:48, Mohammad Islam wrote:
> > trunk/pom.xml, line 137
> > <https://reviews.apache.org/r/3726/diff/3/?file=72016#file72016line137>
> >
> > why do we need it?
we don't. removing.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4786
-----------------------------------------------------------
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3726/
> -----------------------------------------------------------
>
> (Updated 2012-02-02 22:37:20)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
>
> mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
>
> New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
>
> Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
>
> Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
>
>
> This addresses bug OOZIE-667.
> https://issues.apache.org/jira/browse/OOZIE-667
>
>
> Diffs
> -----
>
> trunk/core/pom.xml 1239887
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
> trunk/docs/src/site/twiki/AG_Install.twiki 1239887
> trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
> trunk/examples/pom.xml 1239887
> trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/pom.xml PRE-CREATION
> trunk/pom.xml 1239887
> trunk/sharelib/pom.xml 1239887
> trunk/sharelib/streaming/pom.xml 1239887
> trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
> trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
> trunk/src/main/assemblies/partial-sharelib.xml 1239887
> trunk/webapp/pom.xml 1239887
>
> Diff: https://reviews.apache.org/r/3726/diff
>
>
> Testing
> -------
>
> Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
> On 2012-02-03 03:11:48, Mohammad Islam wrote:
> > trunk/docs/src/site/twiki/DG_QuickStart.twiki, line 51
> > <https://reviews.apache.org/r/3726/diff/3/?file=72007#file72007line51>
> >
> > Same. Is it valid for option # 2.
>
> Alejandro Abdelnur wrote:
> forgot to update the docs. will fix.
This comment stays as it makes sense after the rewording of the previous comment.
> On 2012-02-03 03:11:48, Mohammad Islam wrote:
> > trunk/docs/src/site/twiki/DG_QuickStart.twiki, line 58
> > <https://reviews.apache.org/r/3726/diff/3/?file=72007#file72007line58>
> >
> > So setup script will change too?
>
> Alejandro Abdelnur wrote:
> forgot to update the docs. will fix.
Following the quick start instructions (which state to download the hadooplibs tar.gz) this correction applies. Still I'll make a clarification.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4786
-----------------------------------------------------------
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3726/
> -----------------------------------------------------------
>
> (Updated 2012-02-02 22:37:20)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
>
> mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
>
> New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
>
> Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
>
> Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
>
>
> This addresses bug OOZIE-667.
> https://issues.apache.org/jira/browse/OOZIE-667
>
>
> Diffs
> -----
>
> trunk/core/pom.xml 1239887
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
> trunk/docs/src/site/twiki/AG_Install.twiki 1239887
> trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
> trunk/examples/pom.xml 1239887
> trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/pom.xml PRE-CREATION
> trunk/pom.xml 1239887
> trunk/sharelib/pom.xml 1239887
> trunk/sharelib/streaming/pom.xml 1239887
> trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
> trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
> trunk/src/main/assemblies/partial-sharelib.xml 1239887
> trunk/webapp/pom.xml 1239887
>
> Diff: https://reviews.apache.org/r/3726/diff
>
>
> Testing
> -------
>
> Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4786
-----------------------------------------------------------
trunk/core/pom.xml
<https://reviews.apache.org/r/3726/#comment10519>
is it compile because oozie is building that?
trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/3726/#comment10520>
is the statement correct for our discussed option #2?
trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/3726/#comment10521>
Same. Is it valid for option # 2.
trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/3726/#comment10522>
So setup script will change too?
trunk/pom.xml
<https://reviews.apache.org/r/3726/#comment10518>
why do we need it?
trunk/src/main/assemblies/hadooplib.xml
<https://reviews.apache.org/r/3726/#comment10524>
about the naming: hadooplibs. should we add "oozie" before hand.
trunk/webapp/pom.xml
<https://reviews.apache.org/r/3726/#comment10523>
why it is changed?
- Mohammad
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3726/
> -----------------------------------------------------------
>
> (Updated 2012-02-02 22:37:20)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
>
> mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
>
> New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
>
> Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
>
> Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
>
>
> This addresses bug OOZIE-667.
> https://issues.apache.org/jira/browse/OOZIE-667
>
>
> Diffs
> -----
>
> trunk/core/pom.xml 1239887
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
> trunk/docs/src/site/twiki/AG_Install.twiki 1239887
> trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
> trunk/examples/pom.xml 1239887
> trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/pom.xml PRE-CREATION
> trunk/pom.xml 1239887
> trunk/sharelib/pom.xml 1239887
> trunk/sharelib/streaming/pom.xml 1239887
> trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
> trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
> trunk/src/main/assemblies/partial-sharelib.xml 1239887
> trunk/webapp/pom.xml 1239887
>
> Diff: https://reviews.apache.org/r/3726/diff
>
>
> Testing
> -------
>
> Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4795
-----------------------------------------------------------
Ship it!
+1 after addressing the doc comments for clarifications.
trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/3726/#comment10557>
It is not necessary to use the oozie-bundled hadoop libs. For example, H 0.24.2 is released and we are late or not packaging for some reason. There should be a way, for user to collect the necessary hadoop jars and put it into libext/.
I know this is obvious. Just to mention in some way that user can bring there own hadoop library and copy it into libext.
- Mohammad
On 2012-02-03 05:44:33, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3726/
> -----------------------------------------------------------
>
> (Updated 2012-02-03 05:44:33)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
>
> mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
>
> New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
>
> Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
>
> Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
>
>
> This addresses bug OOZIE-667.
> https://issues.apache.org/jira/browse/OOZIE-667
>
>
> Diffs
> -----
>
> trunk/core/pom.xml 1240004
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1240004
> trunk/docs/src/site/twiki/AG_Install.twiki 1240004
> trunk/docs/src/site/twiki/DG_QuickStart.twiki 1240004
> trunk/examples/pom.xml 1240004
> trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/pom.xml PRE-CREATION
> trunk/pom.xml 1240004
> trunk/sharelib/pom.xml 1240004
> trunk/sharelib/streaming/pom.xml 1240004
> trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
> trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
> trunk/src/main/assemblies/partial-sharelib.xml 1240004
> trunk/webapp/pom.xml 1240004
>
> Diff: https://reviews.apache.org/r/3726/diff
>
>
> Testing
> -------
>
> Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/
-----------------------------------------------------------
(Updated 2012-02-03 05:44:33.292763)
Review request for oozie.
Changes
-------
changes as per Mohammad's comments
Summary
-------
The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
This addresses bug OOZIE-667.
https://issues.apache.org/jira/browse/OOZIE-667
Diffs (updated)
-----
trunk/core/pom.xml 1240004
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1240004
trunk/docs/src/site/twiki/AG_Install.twiki 1240004
trunk/docs/src/site/twiki/DG_QuickStart.twiki 1240004
trunk/examples/pom.xml 1240004
trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/pom.xml PRE-CREATION
trunk/pom.xml 1240004
trunk/sharelib/pom.xml 1240004
trunk/sharelib/streaming/pom.xml 1240004
trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
trunk/src/main/assemblies/partial-sharelib.xml 1240004
trunk/webapp/pom.xml 1240004
Diff: https://reviews.apache.org/r/3726/diff
Testing
-------
Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
Thanks,
Alejandro
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/
-----------------------------------------------------------
(Updated 2012-02-02 22:37:20.758013)
Review request for oozie.
Changes
-------
updated patch that does option #2, it creates a separate tarball for the hadooplibs.
Answering the last question by Mohammad, for each version of hadoop we want to test or have hadooplibs we have to create the corresponding oozie-hadoop artifact. This includes minor&patch releases.
Summary
-------
The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
This addresses bug OOZIE-667.
https://issues.apache.org/jira/browse/OOZIE-667
Diffs (updated)
-----
trunk/core/pom.xml 1239887
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
trunk/docs/src/site/twiki/AG_Install.twiki 1239887
trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
trunk/examples/pom.xml 1239887
trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/pom.xml PRE-CREATION
trunk/pom.xml 1239887
trunk/sharelib/pom.xml 1239887
trunk/sharelib/streaming/pom.xml 1239887
trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
trunk/src/main/assemblies/partial-sharelib.xml 1239887
trunk/webapp/pom.xml 1239887
Diff: https://reviews.apache.org/r/3726/diff
Testing
-------
Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
Thanks,
Alejandro
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
> On 2012-02-02 07:59:13, Mohammad Islam wrote:
> > So it is WIP. right?
> >
> > So we are now bundling the multiple hadoop jars. As the new version of hadoop comes, we will need add a new pom file and associated changes in other pom files.
> > Moreover (in some cases) to support a newly released version we might need to release new oozie version with only new hadoop packages.
> > In addition, oozie tarball size will increase substantially.
> >
> > I think we might need to be little cautious on this.
> >
>
> Alejandro Abdelnur wrote:
> Mohammed,
>
> The patch is not WIP, is ready for commit.
>
> I assume you mean that will be evolving, adding more Hadoop versions as they come, if that is what you meant you are correct.
>
> When a new version of Hadoop comes and we wan to support it in Oozie, we would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it other than setting the desired version from the command line when running Maven '-Dhadoop.version=#####'.
>
> As we stop supporting builds/tests for a particular version of Hadoop we would remove the corresponding hadoop-client/hadoop-test modules from hadooplibs.
>
> Apache projects are Open Source projects and the official distribution of release should be a source tarball. Note that in the source form this does not increase the size of the TAR in a considerable way (just a few KBs for the 2 new POMs).
>
> For convenience, many projects also include binary distributions. You are correct that the more Hadoop versions we have hadooplibs the bigger will be the size of the binary distribution. We have different options for this:
>
> 1* The binary distribution does not include/bundle/produce hadooplibs. The size of Oozie TAR remain will remain constant. This is what we have today, the user is responsible for harvesting the required Hadoop JARs and can use the hadooplib poms as a guidance.
> 2* The binary distribution would be multiple TARs, one for Oozie proper and one for each hadooplib. The size of Oozie TAR will remain constant, the user will have to download the corresponding hadooplib tar.
> 3* Bundle all hadooplibs as part of the binary distribution. (As you've mentioned) The size of the TAR will increase.
>
> Doing #1 the binary TAR is not different from today's.
> Doing #2 the binary TAR is not different from today's, you have additional Oozie hadooplib tars.
> Doing #3 the binary TAR will be bigger from today's.
>
> We can do #1, thus our concerns are addressed and our build easily handles multiple versions of Hadoop.
>
> Thoughts?
>
> Thanks
>
> Alejandro
>
>
>
>
>
>
> Mohammad Islam wrote:
> I prefer both option #1 and option #.
> But Option #2 will be better for easy/quick installation.
> So pls go ahead with option #2.
>
> One question: do we need to add pom for every new hadoop (major/minor) release (although there is no changes in the jar files?
>
>
> Mayank Bansal wrote:
> Hi Alejandro,
>
> For 22 prospective there would not be any client artifact, so we need to do something which will work in both the scenario with and without hadoop client artifacts.
>
> Thanks,
> Mayank
Mayank,
I think you may have misunderstood. This mechanism will handle any version of hadoop, regardless of such version having client artifact or not. Oozie is creating its own set of oozie-client artifacts, one per hadoop version. By doing this we abstract the oozie build from the several different artifacts each version of hadoop may have. If you see the review for OOZIE-672, you'll see that it adds support for Hadoop 0.22.0 to the oozie build.
Hope this clarifies.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3726/
> -----------------------------------------------------------
>
> (Updated 2012-02-02 22:37:20)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
>
> mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
>
> New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
>
> Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
>
> Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
>
>
> This addresses bug OOZIE-667.
> https://issues.apache.org/jira/browse/OOZIE-667
>
>
> Diffs
> -----
>
> trunk/core/pom.xml 1239887
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
> trunk/docs/src/site/twiki/AG_Install.twiki 1239887
> trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
> trunk/examples/pom.xml 1239887
> trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/pom.xml PRE-CREATION
> trunk/pom.xml 1239887
> trunk/sharelib/pom.xml 1239887
> trunk/sharelib/streaming/pom.xml 1239887
> trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
> trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
> trunk/src/main/assemblies/partial-sharelib.xml 1239887
> trunk/webapp/pom.xml 1239887
>
> Diff: https://reviews.apache.org/r/3726/diff
>
>
> Testing
> -------
>
> Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Mayank Bansal <ma...@apache.org>.
> On 2012-02-02 07:59:13, Mohammad Islam wrote:
> > So it is WIP. right?
> >
> > So we are now bundling the multiple hadoop jars. As the new version of hadoop comes, we will need add a new pom file and associated changes in other pom files.
> > Moreover (in some cases) to support a newly released version we might need to release new oozie version with only new hadoop packages.
> > In addition, oozie tarball size will increase substantially.
> >
> > I think we might need to be little cautious on this.
> >
>
> Alejandro Abdelnur wrote:
> Mohammed,
>
> The patch is not WIP, is ready for commit.
>
> I assume you mean that will be evolving, adding more Hadoop versions as they come, if that is what you meant you are correct.
>
> When a new version of Hadoop comes and we wan to support it in Oozie, we would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it other than setting the desired version from the command line when running Maven '-Dhadoop.version=#####'.
>
> As we stop supporting builds/tests for a particular version of Hadoop we would remove the corresponding hadoop-client/hadoop-test modules from hadooplibs.
>
> Apache projects are Open Source projects and the official distribution of release should be a source tarball. Note that in the source form this does not increase the size of the TAR in a considerable way (just a few KBs for the 2 new POMs).
>
> For convenience, many projects also include binary distributions. You are correct that the more Hadoop versions we have hadooplibs the bigger will be the size of the binary distribution. We have different options for this:
>
> 1* The binary distribution does not include/bundle/produce hadooplibs. The size of Oozie TAR remain will remain constant. This is what we have today, the user is responsible for harvesting the required Hadoop JARs and can use the hadooplib poms as a guidance.
> 2* The binary distribution would be multiple TARs, one for Oozie proper and one for each hadooplib. The size of Oozie TAR will remain constant, the user will have to download the corresponding hadooplib tar.
> 3* Bundle all hadooplibs as part of the binary distribution. (As you've mentioned) The size of the TAR will increase.
>
> Doing #1 the binary TAR is not different from today's.
> Doing #2 the binary TAR is not different from today's, you have additional Oozie hadooplib tars.
> Doing #3 the binary TAR will be bigger from today's.
>
> We can do #1, thus our concerns are addressed and our build easily handles multiple versions of Hadoop.
>
> Thoughts?
>
> Thanks
>
> Alejandro
>
>
>
>
>
>
> Mohammad Islam wrote:
> I prefer both option #1 and option #.
> But Option #2 will be better for easy/quick installation.
> So pls go ahead with option #2.
>
> One question: do we need to add pom for every new hadoop (major/minor) release (although there is no changes in the jar files?
>
Hi Alejandro,
For 22 prospective there would not be any client artifact, so we need to do something which will work in both the scenario with and without hadoop client artifacts.
Thanks,
Mayank
- Mayank
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3726/
> -----------------------------------------------------------
>
> (Updated 2012-02-02 22:37:20)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
>
> mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
>
> New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
>
> Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
>
> Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
>
>
> This addresses bug OOZIE-667.
> https://issues.apache.org/jira/browse/OOZIE-667
>
>
> Diffs
> -----
>
> trunk/core/pom.xml 1239887
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
> trunk/docs/src/site/twiki/AG_Install.twiki 1239887
> trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
> trunk/examples/pom.xml 1239887
> trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/pom.xml PRE-CREATION
> trunk/pom.xml 1239887
> trunk/sharelib/pom.xml 1239887
> trunk/sharelib/streaming/pom.xml 1239887
> trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
> trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
> trunk/src/main/assemblies/partial-sharelib.xml 1239887
> trunk/webapp/pom.xml 1239887
>
> Diff: https://reviews.apache.org/r/3726/diff
>
>
> Testing
> -------
>
> Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
> On 2012-02-02 07:59:13, Mohammad Islam wrote:
> > So it is WIP. right?
> >
> > So we are now bundling the multiple hadoop jars. As the new version of hadoop comes, we will need add a new pom file and associated changes in other pom files.
> > Moreover (in some cases) to support a newly released version we might need to release new oozie version with only new hadoop packages.
> > In addition, oozie tarball size will increase substantially.
> >
> > I think we might need to be little cautious on this.
> >
Mohammed,
The patch is not WIP, is ready for commit.
I assume you mean that will be evolving, adding more Hadoop versions as they come, if that is what you meant you are correct.
When a new version of Hadoop comes and we wan to support it in Oozie, we would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it other than setting the desired version from the command line when running Maven '-Dhadoop.version=#####'.
As we stop supporting builds/tests for a particular version of Hadoop we would remove the corresponding hadoop-client/hadoop-test modules from hadooplibs.
Apache projects are Open Source projects and the official distribution of release should be a source tarball. Note that in the source form this does not increase the size of the TAR in a considerable way (just a few KBs for the 2 new POMs).
For convenience, many projects also include binary distributions. You are correct that the more Hadoop versions we have hadooplibs the bigger will be the size of the binary distribution. We have different options for this:
1* The binary distribution does not include/bundle/produce hadooplibs. The size of Oozie TAR remain will remain constant. This is what we have today, the user is responsible for harvesting the required Hadoop JARs and can use the hadooplib poms as a guidance.
2* The binary distribution would be multiple TARs, one for Oozie proper and one for each hadooplib. The size of Oozie TAR will remain constant, the user will have to download the corresponding hadooplib tar.
3* Bundle all hadooplibs as part of the binary distribution. (As you've mentioned) The size of the TAR will increase.
Doing #1 the binary TAR is not different from today's.
Doing #2 the binary TAR is not different from today's, you have additional Oozie hadooplib tars.
Doing #3 the binary TAR will be bigger from today's.
We can do #1, thus our concerns are addressed and our build easily handles multiple versions of Hadoop.
Thoughts?
Thanks
Alejandro
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
On 2012-02-01 20:18:21, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3726/
> -----------------------------------------------------------
>
> (Updated 2012-02-01 20:18:21)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
>
> mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
>
> New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
>
> Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
>
> Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
>
>
> This addresses bug OOZIE-667.
> https://issues.apache.org/jira/browse/OOZIE-667
>
>
> Diffs
> -----
>
> trunk/core/pom.xml 1239282
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282
> trunk/docs/src/site/twiki/AG_Install.twiki 1239282
> trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282
> trunk/examples/pom.xml 1239282
> trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/pom.xml PRE-CREATION
> trunk/pom.xml 1239282
> trunk/sharelib/pom.xml 1239282
> trunk/sharelib/streaming/pom.xml 1239282
> trunk/src/main/assemblies/distro.xml 1239282
> trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
> trunk/src/main/assemblies/partial-sharelib.xml 1239282
> trunk/webapp/pom.xml 1239282
>
> Diff: https://reviews.apache.org/r/3726/diff
>
>
> Testing
> -------
>
> Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Mohammad Islam <mi...@yahoo.com>.
> On 2012-02-02 07:59:13, Mohammad Islam wrote:
> > So it is WIP. right?
> >
> > So we are now bundling the multiple hadoop jars. As the new version of hadoop comes, we will need add a new pom file and associated changes in other pom files.
> > Moreover (in some cases) to support a newly released version we might need to release new oozie version with only new hadoop packages.
> > In addition, oozie tarball size will increase substantially.
> >
> > I think we might need to be little cautious on this.
> >
>
> Alejandro Abdelnur wrote:
> Mohammed,
>
> The patch is not WIP, is ready for commit.
>
> I assume you mean that will be evolving, adding more Hadoop versions as they come, if that is what you meant you are correct.
>
> When a new version of Hadoop comes and we wan to support it in Oozie, we would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it other than setting the desired version from the command line when running Maven '-Dhadoop.version=#####'.
>
> As we stop supporting builds/tests for a particular version of Hadoop we would remove the corresponding hadoop-client/hadoop-test modules from hadooplibs.
>
> Apache projects are Open Source projects and the official distribution of release should be a source tarball. Note that in the source form this does not increase the size of the TAR in a considerable way (just a few KBs for the 2 new POMs).
>
> For convenience, many projects also include binary distributions. You are correct that the more Hadoop versions we have hadooplibs the bigger will be the size of the binary distribution. We have different options for this:
>
> 1* The binary distribution does not include/bundle/produce hadooplibs. The size of Oozie TAR remain will remain constant. This is what we have today, the user is responsible for harvesting the required Hadoop JARs and can use the hadooplib poms as a guidance.
> 2* The binary distribution would be multiple TARs, one for Oozie proper and one for each hadooplib. The size of Oozie TAR will remain constant, the user will have to download the corresponding hadooplib tar.
> 3* Bundle all hadooplibs as part of the binary distribution. (As you've mentioned) The size of the TAR will increase.
>
> Doing #1 the binary TAR is not different from today's.
> Doing #2 the binary TAR is not different from today's, you have additional Oozie hadooplib tars.
> Doing #3 the binary TAR will be bigger from today's.
>
> We can do #1, thus our concerns are addressed and our build easily handles multiple versions of Hadoop.
>
> Thoughts?
>
> Thanks
>
> Alejandro
>
>
>
>
>
I prefer both option #1 and option #.
But Option #2 will be better for easy/quick installation.
So pls go ahead with option #2.
One question: do we need to add pom for every new hadoop (major/minor) release (although there is no changes in the jar files?
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
On 2012-02-01 20:18:21, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3726/
> -----------------------------------------------------------
>
> (Updated 2012-02-01 20:18:21)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
>
> mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
>
> New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
>
> Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
>
> Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
>
>
> This addresses bug OOZIE-667.
> https://issues.apache.org/jira/browse/OOZIE-667
>
>
> Diffs
> -----
>
> trunk/core/pom.xml 1239282
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282
> trunk/docs/src/site/twiki/AG_Install.twiki 1239282
> trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282
> trunk/examples/pom.xml 1239282
> trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/pom.xml PRE-CREATION
> trunk/pom.xml 1239282
> trunk/sharelib/pom.xml 1239282
> trunk/sharelib/streaming/pom.xml 1239282
> trunk/src/main/assemblies/distro.xml 1239282
> trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
> trunk/src/main/assemblies/partial-sharelib.xml 1239282
> trunk/webapp/pom.xml 1239282
>
> Diff: https://reviews.apache.org/r/3726/diff
>
>
> Testing
> -------
>
> Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-667 Change the way Oozie brings in Hadoop JARs
into the build
Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
So it is WIP. right?
So we are now bundling the multiple hadoop jars. As the new version of hadoop comes, we will need add a new pom file and associated changes in other pom files.
Moreover (in some cases) to support a newly released version we might need to release new oozie version with only new hadoop packages.
In addition, oozie tarball size will increase substantially.
I think we might need to be little cautious on this.
- Mohammad
On 2012-02-01 20:18:21, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3726/
> -----------------------------------------------------------
>
> (Updated 2012-02-01 20:18:21)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
>
> mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
>
> New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
>
> Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
>
> Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
>
>
> This addresses bug OOZIE-667.
> https://issues.apache.org/jira/browse/OOZIE-667
>
>
> Diffs
> -----
>
> trunk/core/pom.xml 1239282
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282
> trunk/docs/src/site/twiki/AG_Install.twiki 1239282
> trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282
> trunk/examples/pom.xml 1239282
> trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
> trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
> trunk/hadooplibs/pom.xml PRE-CREATION
> trunk/pom.xml 1239282
> trunk/sharelib/pom.xml 1239282
> trunk/sharelib/streaming/pom.xml 1239282
> trunk/src/main/assemblies/distro.xml 1239282
> trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
> trunk/src/main/assemblies/partial-sharelib.xml 1239282
> trunk/webapp/pom.xml 1239282
>
> Diff: https://reviews.apache.org/r/3726/diff
>
>
> Testing
> -------
>
> Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
>
>
> Thanks,
>
> Alejandro
>
>