You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Alejandro Abdelnur (Created) (JIRA)" <ji...@apache.org> on 2012/01/31 20:44:10 UTC
[jira] [Created] (OOZIE-667) Change the way Oozie brings in Hadoop
JARs into the build
Change the way Oozie brings in Hadoop JARs into the build
---------------------------------------------------------
Key: OOZIE-667
URL: https://issues.apache.org/jira/browse/OOZIE-667
Project: Oozie
Issue Type: Improvement
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Currently we are using profiles with several exclussions and they are through out the POMs.
A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199387#comment-13199387 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
bq. On 2012-02-02 07:59:13, Mohammad Islam wrote:
bq. > So it is WIP. right?
bq. >
bq. > So we are now bundling the multiple hadoop jars. As the new version of hadoop comes, we will need add a new pom file and associated changes in other pom files.
bq. > Moreover (in some cases) to support a newly released version we might need to release new oozie version with only new hadoop packages.
bq. > In addition, oozie tarball size will increase substantially.
bq. >
bq. > I think we might need to be little cautious on this.
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Mohammed,
bq.
bq. The patch is not WIP, is ready for commit.
bq.
bq. I assume you mean that will be evolving, adding more Hadoop versions as they come, if that is what you meant you are correct.
bq.
bq. When a new version of Hadoop comes and we wan to support it in Oozie, we would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it other than setting the desired version from the command line when running Maven '-Dhadoop.version=#####'.
bq.
bq. As we stop supporting builds/tests for a particular version of Hadoop we would remove the corresponding hadoop-client/hadoop-test modules from hadooplibs.
bq.
bq. Apache projects are Open Source projects and the official distribution of release should be a source tarball. Note that in the source form this does not increase the size of the TAR in a considerable way (just a few KBs for the 2 new POMs).
bq.
bq. For convenience, many projects also include binary distributions. You are correct that the more Hadoop versions we have hadooplibs the bigger will be the size of the binary distribution. We have different options for this:
bq.
bq. 1* The binary distribution does not include/bundle/produce hadooplibs. The size of Oozie TAR remain will remain constant. This is what we have today, the user is responsible for harvesting the required Hadoop JARs and can use the hadooplib poms as a guidance.
bq. 2* The binary distribution would be multiple TARs, one for Oozie proper and one for each hadooplib. The size of Oozie TAR will remain constant, the user will have to download the corresponding hadooplib tar.
bq. 3* Bundle all hadooplibs as part of the binary distribution. (As you've mentioned) The size of the TAR will increase.
bq.
bq. Doing #1 the binary TAR is not different from today's.
bq. Doing #2 the binary TAR is not different from today's, you have additional Oozie hadooplib tars.
bq. Doing #3 the binary TAR will be bigger from today's.
bq.
bq. We can do #1, thus our concerns are addressed and our build easily handles multiple versions of Hadoop.
bq.
bq. Thoughts?
bq.
bq. Thanks
bq.
bq. Alejandro
bq.
bq.
bq.
bq.
bq.
bq.
bq. Mohammad Islam wrote:
bq. I prefer both option #1 and option #.
bq. But Option #2 will be better for easy/quick installation.
bq. So pls go ahead with option #2.
bq.
bq. One question: do we need to add pom for every new hadoop (major/minor) release (although there is no changes in the jar files?
bq.
bq.
bq. Mayank Bansal wrote:
bq. Hi Alejandro,
bq.
bq. For 22 prospective there would not be any client artifact, so we need to do something which will work in both the scenario with and without hadoop client artifacts.
bq.
bq. Thanks,
bq. Mayank
Mayank,
I think you may have misunderstood. This mechanism will handle any version of hadoop, regardless of such version having client artifact or not. Oozie is creating its own set of oozie-client artifacts, one per hadoop version. By doing this we abstract the oozie build from the several different artifacts each version of hadoop may have. If you see the review for OOZIE-672, you'll see that it adds support for Hadoop 0.22.0 to the oozie build.
Hope this clarifies.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-02 22:37:20)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1239887
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1239887
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
bq. trunk/examples/pom.xml 1239887
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1239887
bq. trunk/sharelib/pom.xml 1239887
bq. trunk/sharelib/streaming/pom.xml 1239887
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1239887
bq. trunk/webapp/pom.xml 1239887
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OOZIE-667) Change the way Oozie brings in Hadoop
JARs into the build
Posted by "Alejandro Abdelnur (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur resolved OOZIE-667.
--------------------------------------
Resolution: Fixed
Committed to trunk
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198113#comment-13198113 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/
-----------------------------------------------------------
Review request for oozie.
Summary
-------
The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
This addresses bug OOZIE-667.
https://issues.apache.org/jira/browse/OOZIE-667
Diffs
-----
trunk/core/pom.xml 1239282
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282
trunk/docs/src/site/twiki/AG_Install.twiki 1239282
trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282
trunk/examples/pom.xml 1239282
trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/pom.xml PRE-CREATION
trunk/pom.xml 1239282
trunk/sharelib/pom.xml 1239282
trunk/sharelib/streaming/pom.xml 1239282
trunk/src/main/assemblies/distro.xml 1239282
trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
trunk/src/main/assemblies/partial-sharelib.xml 1239282
trunk/webapp/pom.xml 1239282
Diff: https://reviews.apache.org/r/3726/diff
Testing
-------
Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
Thanks,
Alejandro
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199536#comment-13199536 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/
-----------------------------------------------------------
(Updated 2012-02-03 05:44:33.292763)
Review request for oozie.
Changes
-------
changes as per Mohammad's comments
Summary
-------
The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
This addresses bug OOZIE-667.
https://issues.apache.org/jira/browse/OOZIE-667
Diffs (updated)
-----
trunk/core/pom.xml 1240004
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1240004
trunk/docs/src/site/twiki/AG_Install.twiki 1240004
trunk/docs/src/site/twiki/DG_QuickStart.twiki 1240004
trunk/examples/pom.xml 1240004
trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/pom.xml PRE-CREATION
trunk/pom.xml 1240004
trunk/sharelib/pom.xml 1240004
trunk/sharelib/streaming/pom.xml 1240004
trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
trunk/src/main/assemblies/partial-sharelib.xml 1240004
trunk/webapp/pom.xml 1240004
Diff: https://reviews.apache.org/r/3726/diff
Testing
-------
Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
Thanks,
Alejandro
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-667) Change the way Oozie brings in Hadoop
JARs into the build
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated OOZIE-667:
-------------------------------------
Fix Version/s: 3.2.0
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 3.2.0
>
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198592#comment-13198592 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
So it is WIP. right?
So we are now bundling the multiple hadoop jars. As the new version of hadoop comes, we will need add a new pom file and associated changes in other pom files.
Moreover (in some cases) to support a newly released version we might need to release new oozie version with only new hadoop packages.
In addition, oozie tarball size will increase substantially.
I think we might need to be little cautious on this.
- Mohammad
On 2012-02-01 20:18:21, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-01 20:18:21)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1239282
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1239282
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282
bq. trunk/examples/pom.xml 1239282
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1239282
bq. trunk/sharelib/pom.xml 1239282
bq. trunk/sharelib/streaming/pom.xml 1239282
bq. trunk/src/main/assemblies/distro.xml 1239282
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1239282
bq. trunk/webapp/pom.xml 1239282
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198901#comment-13198901 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
bq. On 2012-02-02 07:59:13, Mohammad Islam wrote:
bq. > So it is WIP. right?
bq. >
bq. > So we are now bundling the multiple hadoop jars. As the new version of hadoop comes, we will need add a new pom file and associated changes in other pom files.
bq. > Moreover (in some cases) to support a newly released version we might need to release new oozie version with only new hadoop packages.
bq. > In addition, oozie tarball size will increase substantially.
bq. >
bq. > I think we might need to be little cautious on this.
bq. >
Mohammed,
The patch is not WIP, is ready for commit.
I assume you mean that will be evolving, adding more Hadoop versions as they come, if that is what you meant you are correct.
When a new version of Hadoop comes and we wan to support it in Oozie, we would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it other than setting the desired version from the command line when running Maven '-Dhadoop.version=#####'.
As we stop supporting builds/tests for a particular version of Hadoop we would remove the corresponding hadoop-client/hadoop-test modules from hadooplibs.
Apache projects are Open Source projects and the official distribution of release should be a source tarball. Note that in the source form this does not increase the size of the TAR in a considerable way (just a few KBs for the 2 new POMs).
For convenience, many projects also include binary distributions. You are correct that the more Hadoop versions we have hadooplibs the bigger will be the size of the binary distribution. We have different options for this:
1* The binary distribution does not include/bundle/produce hadooplibs. The size of Oozie TAR remain will remain constant. This is what we have today, the user is responsible for harvesting the required Hadoop JARs and can use the hadooplib poms as a guidance.
2* The binary distribution would be multiple TARs, one for Oozie proper and one for each hadooplib. The size of Oozie TAR will remain constant, the user will have to download the corresponding hadooplib tar.
3* Bundle all hadooplibs as part of the binary distribution. (As you've mentioned) The size of the TAR will increase.
Doing #1 the binary TAR is not different from today's.
Doing #2 the binary TAR is not different from today's, you have additional Oozie hadooplib tars.
Doing #3 the binary TAR will be bigger from today's.
We can do #1, thus our concerns are addressed and our build easily handles multiple versions of Hadoop.
Thoughts?
Thanks
Alejandro
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
On 2012-02-01 20:18:21, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-01 20:18:21)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1239282
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1239282
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282
bq. trunk/examples/pom.xml 1239282
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1239282
bq. trunk/sharelib/pom.xml 1239282
bq. trunk/sharelib/streaming/pom.xml 1239282
bq. trunk/src/main/assemblies/distro.xml 1239282
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1239282
bq. trunk/webapp/pom.xml 1239282
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199530#comment-13199530 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
bq. On 2012-02-03 03:11:48, Mohammad Islam wrote:
bq. > trunk/pom.xml, line 137
bq. > <https://reviews.apache.org/r/3726/diff/3/?file=72016#file72016line137>
bq. >
bq. > why do we need it?
we don't. removing.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4786
-----------------------------------------------------------
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-02 22:37:20)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1239887
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1239887
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
bq. trunk/examples/pom.xml 1239887
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1239887
bq. trunk/sharelib/pom.xml 1239887
bq. trunk/sharelib/streaming/pom.xml 1239887
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1239887
bq. trunk/webapp/pom.xml 1239887
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199261#comment-13199261 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
bq. On 2012-02-02 07:59:13, Mohammad Islam wrote:
bq. > So it is WIP. right?
bq. >
bq. > So we are now bundling the multiple hadoop jars. As the new version of hadoop comes, we will need add a new pom file and associated changes in other pom files.
bq. > Moreover (in some cases) to support a newly released version we might need to release new oozie version with only new hadoop packages.
bq. > In addition, oozie tarball size will increase substantially.
bq. >
bq. > I think we might need to be little cautious on this.
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Mohammed,
bq.
bq. The patch is not WIP, is ready for commit.
bq.
bq. I assume you mean that will be evolving, adding more Hadoop versions as they come, if that is what you meant you are correct.
bq.
bq. When a new version of Hadoop comes and we wan to support it in Oozie, we would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it other than setting the desired version from the command line when running Maven '-Dhadoop.version=#####'.
bq.
bq. As we stop supporting builds/tests for a particular version of Hadoop we would remove the corresponding hadoop-client/hadoop-test modules from hadooplibs.
bq.
bq. Apache projects are Open Source projects and the official distribution of release should be a source tarball. Note that in the source form this does not increase the size of the TAR in a considerable way (just a few KBs for the 2 new POMs).
bq.
bq. For convenience, many projects also include binary distributions. You are correct that the more Hadoop versions we have hadooplibs the bigger will be the size of the binary distribution. We have different options for this:
bq.
bq. 1* The binary distribution does not include/bundle/produce hadooplibs. The size of Oozie TAR remain will remain constant. This is what we have today, the user is responsible for harvesting the required Hadoop JARs and can use the hadooplib poms as a guidance.
bq. 2* The binary distribution would be multiple TARs, one for Oozie proper and one for each hadooplib. The size of Oozie TAR will remain constant, the user will have to download the corresponding hadooplib tar.
bq. 3* Bundle all hadooplibs as part of the binary distribution. (As you've mentioned) The size of the TAR will increase.
bq.
bq. Doing #1 the binary TAR is not different from today's.
bq. Doing #2 the binary TAR is not different from today's, you have additional Oozie hadooplib tars.
bq. Doing #3 the binary TAR will be bigger from today's.
bq.
bq. We can do #1, thus our concerns are addressed and our build easily handles multiple versions of Hadoop.
bq.
bq. Thoughts?
bq.
bq. Thanks
bq.
bq. Alejandro
bq.
bq.
bq.
bq.
bq.
I prefer both option #1 and option #.
But Option #2 will be better for easy/quick installation.
So pls go ahead with option #2.
One question: do we need to add pom for every new hadoop (major/minor) release (although there is no changes in the jar files?
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
On 2012-02-01 20:18:21, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-01 20:18:21)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1239282
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1239282
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282
bq. trunk/examples/pom.xml 1239282
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1239282
bq. trunk/sharelib/pom.xml 1239282
bq. trunk/sharelib/streaming/pom.xml 1239282
bq. trunk/src/main/assemblies/distro.xml 1239282
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1239282
bq. trunk/webapp/pom.xml 1239282
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199493#comment-13199493 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4786
-----------------------------------------------------------
trunk/core/pom.xml
<https://reviews.apache.org/r/3726/#comment10519>
is it compile because oozie is building that?
trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/3726/#comment10520>
is the statement correct for our discussed option #2?
trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/3726/#comment10521>
Same. Is it valid for option # 2.
trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/3726/#comment10522>
So setup script will change too?
trunk/pom.xml
<https://reviews.apache.org/r/3726/#comment10518>
why do we need it?
trunk/src/main/assemblies/hadooplib.xml
<https://reviews.apache.org/r/3726/#comment10524>
about the naming: hadooplibs. should we add "oozie" before hand.
trunk/webapp/pom.xml
<https://reviews.apache.org/r/3726/#comment10523>
why it is changed?
- Mohammad
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-02 22:37:20)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1239887
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1239887
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
bq. trunk/examples/pom.xml 1239887
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1239887
bq. trunk/sharelib/pom.xml 1239887
bq. trunk/sharelib/streaming/pom.xml 1239887
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1239887
bq. trunk/webapp/pom.xml 1239887
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199306#comment-13199306 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/
-----------------------------------------------------------
(Updated 2012-02-02 22:37:20.758013)
Review request for oozie.
Changes
-------
updated patch that does option #2, it creates a separate tarball for the hadooplibs.
Answering the last question by Mohammad, for each version of hadoop we want to test or have hadooplibs we have to create the corresponding oozie-hadoop artifact. This includes minor&patch releases.
Summary
-------
The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
This addresses bug OOZIE-667.
https://issues.apache.org/jira/browse/OOZIE-667
Diffs (updated)
-----
trunk/core/pom.xml 1239887
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
trunk/docs/src/site/twiki/AG_Install.twiki 1239887
trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
trunk/examples/pom.xml 1239887
trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
trunk/hadooplibs/pom.xml PRE-CREATION
trunk/pom.xml 1239887
trunk/sharelib/pom.xml 1239887
trunk/sharelib/streaming/pom.xml 1239887
trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
trunk/src/main/assemblies/partial-sharelib.xml 1239887
trunk/webapp/pom.xml 1239887
Diff: https://reviews.apache.org/r/3726/diff
Testing
-------
Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
Thanks,
Alejandro
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199589#comment-13199589 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4795
-----------------------------------------------------------
Ship it!
+1 after addressing the doc comments for clarifications.
trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/3726/#comment10557>
It is not necessary to use the oozie-bundled hadoop libs. For example, H 0.24.2 is released and we are late or not packaging for some reason. There should be a way, for user to collect the necessary hadoop jars and put it into libext/.
I know this is obvious. Just to mention in some way that user can bring there own hadoop library and copy it into libext.
- Mohammad
On 2012-02-03 05:44:33, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-03 05:44:33)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1240004
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1240004
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1240004
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1240004
bq. trunk/examples/pom.xml 1240004
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1240004
bq. trunk/sharelib/pom.xml 1240004
bq. trunk/sharelib/streaming/pom.xml 1240004
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1240004
bq. trunk/webapp/pom.xml 1240004
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199532#comment-13199532 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
bq. On 2012-02-03 03:11:48, Mohammad Islam wrote:
bq. > trunk/docs/src/site/twiki/DG_QuickStart.twiki, line 51
bq. > <https://reviews.apache.org/r/3726/diff/3/?file=72007#file72007line51>
bq. >
bq. > Same. Is it valid for option # 2.
bq.
bq. Alejandro Abdelnur wrote:
bq. forgot to update the docs. will fix.
This comment stays as it makes sense after the rewording of the previous comment.
bq. On 2012-02-03 03:11:48, Mohammad Islam wrote:
bq. > trunk/docs/src/site/twiki/DG_QuickStart.twiki, line 58
bq. > <https://reviews.apache.org/r/3726/diff/3/?file=72007#file72007line58>
bq. >
bq. > So setup script will change too?
bq.
bq. Alejandro Abdelnur wrote:
bq. forgot to update the docs. will fix.
Following the quick start instructions (which state to download the hadooplibs tar.gz) this correction applies. Still I'll make a clarification.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4786
-----------------------------------------------------------
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-02 22:37:20)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1239887
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1239887
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
bq. trunk/examples/pom.xml 1239887
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1239887
bq. trunk/sharelib/pom.xml 1239887
bq. trunk/sharelib/streaming/pom.xml 1239887
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1239887
bq. trunk/webapp/pom.xml 1239887
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199378#comment-13199378 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
bq. On 2012-02-02 07:59:13, Mohammad Islam wrote:
bq. > So it is WIP. right?
bq. >
bq. > So we are now bundling the multiple hadoop jars. As the new version of hadoop comes, we will need add a new pom file and associated changes in other pom files.
bq. > Moreover (in some cases) to support a newly released version we might need to release new oozie version with only new hadoop packages.
bq. > In addition, oozie tarball size will increase substantially.
bq. >
bq. > I think we might need to be little cautious on this.
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Mohammed,
bq.
bq. The patch is not WIP, is ready for commit.
bq.
bq. I assume you mean that will be evolving, adding more Hadoop versions as they come, if that is what you meant you are correct.
bq.
bq. When a new version of Hadoop comes and we wan to support it in Oozie, we would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it other than setting the desired version from the command line when running Maven '-Dhadoop.version=#####'.
bq.
bq. As we stop supporting builds/tests for a particular version of Hadoop we would remove the corresponding hadoop-client/hadoop-test modules from hadooplibs.
bq.
bq. Apache projects are Open Source projects and the official distribution of release should be a source tarball. Note that in the source form this does not increase the size of the TAR in a considerable way (just a few KBs for the 2 new POMs).
bq.
bq. For convenience, many projects also include binary distributions. You are correct that the more Hadoop versions we have hadooplibs the bigger will be the size of the binary distribution. We have different options for this:
bq.
bq. 1* The binary distribution does not include/bundle/produce hadooplibs. The size of Oozie TAR remain will remain constant. This is what we have today, the user is responsible for harvesting the required Hadoop JARs and can use the hadooplib poms as a guidance.
bq. 2* The binary distribution would be multiple TARs, one for Oozie proper and one for each hadooplib. The size of Oozie TAR will remain constant, the user will have to download the corresponding hadooplib tar.
bq. 3* Bundle all hadooplibs as part of the binary distribution. (As you've mentioned) The size of the TAR will increase.
bq.
bq. Doing #1 the binary TAR is not different from today's.
bq. Doing #2 the binary TAR is not different from today's, you have additional Oozie hadooplib tars.
bq. Doing #3 the binary TAR will be bigger from today's.
bq.
bq. We can do #1, thus our concerns are addressed and our build easily handles multiple versions of Hadoop.
bq.
bq. Thoughts?
bq.
bq. Thanks
bq.
bq. Alejandro
bq.
bq.
bq.
bq.
bq.
bq.
bq. Mohammad Islam wrote:
bq. I prefer both option #1 and option #.
bq. But Option #2 will be better for easy/quick installation.
bq. So pls go ahead with option #2.
bq.
bq. One question: do we need to add pom for every new hadoop (major/minor) release (although there is no changes in the jar files?
bq.
Hi Alejandro,
For 22 prospective there would not be any client artifact, so we need to do something which will work in both the scenario with and without hadoop client artifacts.
Thanks,
Mayank
- Mayank
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-02 22:37:20)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1239887
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1239887
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
bq. trunk/examples/pom.xml 1239887
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1239887
bq. trunk/sharelib/pom.xml 1239887
bq. trunk/sharelib/streaming/pom.xml 1239887
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1239887
bq. trunk/webapp/pom.xml 1239887
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-667) Change the way Oozie brings in
Hadoop JARs into the build
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199526#comment-13199526 ]
jiraposter@reviews.apache.org commented on OOZIE-667:
-----------------------------------------------------
bq. On 2012-02-03 03:11:48, Mohammad Islam wrote:
bq. > trunk/docs/src/site/twiki/DG_QuickStart.twiki, line 39
bq. > <https://reviews.apache.org/r/3726/diff/3/?file=72007#file72007line39>
bq. >
bq. > is the statement correct for our discussed option #2?
forgot to update the docs. will fix.
bq. On 2012-02-03 03:11:48, Mohammad Islam wrote:
bq. > trunk/core/pom.xml, line 45
bq. > <https://reviews.apache.org/r/3726/diff/3/?file=72004#file72004line45>
bq. >
bq. > is it compile because oozie is building that?
this should be 'provided' i was playing around in my IDE. will fix
bq. On 2012-02-03 03:11:48, Mohammad Islam wrote:
bq. > trunk/docs/src/site/twiki/DG_QuickStart.twiki, line 51
bq. > <https://reviews.apache.org/r/3726/diff/3/?file=72007#file72007line51>
bq. >
bq. > Same. Is it valid for option # 2.
forgot to update the docs. will fix.
bq. On 2012-02-03 03:11:48, Mohammad Islam wrote:
bq. > trunk/docs/src/site/twiki/DG_QuickStart.twiki, line 58
bq. > <https://reviews.apache.org/r/3726/diff/3/?file=72007#file72007line58>
bq. >
bq. > So setup script will change too?
forgot to update the docs. will fix.
bq. On 2012-02-03 03:11:48, Mohammad Islam wrote:
bq. > trunk/src/main/assemblies/hadooplib.xml, line 32
bq. > <https://reviews.apache.org/r/3726/diff/3/?file=72019#file72019line32>
bq. >
bq. > about the naming: hadooplibs. should we add "oozie" before hand.
bq. >
having 2 TAR, oozie proper TAR and hadooplib TAR, if you expand both in the same location, you'll get hadooplibs/ in the oozie-VERSION directory.
bq. On 2012-02-03 03:11:48, Mohammad Islam wrote:
bq. > trunk/webapp/pom.xml, line 69
bq. > <https://reviews.apache.org/r/3726/diff/3/?file=72022#file72022line69>
bq. >
bq. > why it is changed?
because it was a wrong groupId, found this out while tuning the exclusions for the hadooplibs.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4786
-----------------------------------------------------------
On 2012-02-02 22:37:20, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-02 22:37:20)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a hadoop-client artifact (even for already released versions), still we'll need the corresponding hadooplibs module to be able to use the assembly (as it is done as part of this patch) to pull into the oozie distro the JARs for the supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't have to be aware of the JARs deps of different versions of Hadoop or of hadoop JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1239887
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239887
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1239887
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239887
bq. trunk/examples/pom.xml 1239887
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1239887
bq. trunk/sharelib/pom.xml 1239887
bq. trunk/sharelib/streaming/pom.xml 1239887
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/hadooplibs.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1239887
bq. trunk/webapp/pom.xml 1239887
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs would be bundled with the release. Then it would be much easier for oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira