You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Alejandro Abdelnur <tu...@cloudera.com> on 2011/11/09 02:36:07 UTC
Review Request: Make Oozie compile against Hadoop 0.23
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/
-----------------------------------------------------------
Review request for oozie, Tom White and Roman Shaposhnik.
Summary
-------
Make Oozie compile against Hadoop 0.23.
It depends on a few outstanding Hadoop JIRAs.
Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
This patch is a WIP
This addresses bug OOZIE-565.
https://issues.apache.org/jira/browse/OOZIE-565
Diffs
-----
/trunk/core/pom.xml 1199584
/trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1199584
/trunk/examples/pom.xml 1199584
/trunk/pom.xml 1199584
/trunk/sharelib/pom.xml 1199584
Diff: https://reviews.apache.org/r/2778/diff
Testing
-------
Thanks,
Alejandro
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
> On 2011-11-10 01:48:32, Mohammad Islam wrote:
> > Looks very good.
> > Want to clarify that jar injection will be done during runtime. not during packaging. right?
> >
> > About running job successfully through Oozie: Mona has done it. But it needs some code changes in H 0.23. Not sure it was committed to hadoop trunk yet.
> >
JAR Injection, as until now can be done at build time, by using the -DincludeHadoopJars option or after installation.
Would be possible to share Mona's findings?
Given that using 0.23 is an optional profile, I'd like to commit this to trunk to facilitate work for Bigtop.
Thanks.
Alejandro
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/#review3146
-----------------------------------------------------------
On 2011-11-10 00:06:30, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2778/
> -----------------------------------------------------------
>
> (Updated 2011-11-10 00:06:30)
>
>
> Review request for oozie, Tom White and Roman Shaposhnik.
>
>
> Summary
> -------
>
> Make Oozie compile against Hadoop 0.23.
>
> It depends on a few outstanding Hadoop JIRAs.
>
> Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
>
> This patch is a WIP
>
>
> This addresses bug OOZIE-565.
> https://issues.apache.org/jira/browse/OOZIE-565
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1200045
> /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1200045
> /trunk/examples/pom.xml 1200045
> /trunk/pom.xml 1200045
> /trunk/sharelib/pom.xml 1200045
> /trunk/webapp/pom.xml 1200045
>
> Diff: https://reviews.apache.org/r/2778/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/#review3146
-----------------------------------------------------------
Looks very good.
Want to clarify that jar injection will be done during runtime. not during packaging. right?
About running job successfully through Oozie: Mona has done it. But it needs some code changes in H 0.23. Not sure it was committed to hadoop trunk yet.
- Mohammad
On 2011-11-10 00:06:30, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2778/
> -----------------------------------------------------------
>
> (Updated 2011-11-10 00:06:30)
>
>
> Review request for oozie, Tom White and Roman Shaposhnik.
>
>
> Summary
> -------
>
> Make Oozie compile against Hadoop 0.23.
>
> It depends on a few outstanding Hadoop JIRAs.
>
> Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
>
> This patch is a WIP
>
>
> This addresses bug OOZIE-565.
> https://issues.apache.org/jira/browse/OOZIE-565
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1200045
> /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1200045
> /trunk/examples/pom.xml 1200045
> /trunk/pom.xml 1200045
> /trunk/sharelib/pom.xml 1200045
> /trunk/webapp/pom.xml 1200045
>
> Diff: https://reviews.apache.org/r/2778/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
> On 2011-11-16 23:37:15, Angelo K. Huang wrote:
> > I noticed that you have used hadoop.jars.scope at webapp. Why not use it at other places as well?
because here is the place we decide to bundle or not Hadoop dependencies in the distribution based on the includeHadoopJar profile, for other places is not required.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/#review3308
-----------------------------------------------------------
On 2011-11-10 00:06:30, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2778/
> -----------------------------------------------------------
>
> (Updated 2011-11-10 00:06:30)
>
>
> Review request for oozie, Tom White and Roman Shaposhnik.
>
>
> Summary
> -------
>
> Make Oozie compile against Hadoop 0.23.
>
> It depends on a few outstanding Hadoop JIRAs.
>
> Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
>
> This patch is a WIP
>
>
> This addresses bug OOZIE-565.
> https://issues.apache.org/jira/browse/OOZIE-565
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1200045
> /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1200045
> /trunk/examples/pom.xml 1200045
> /trunk/pom.xml 1200045
> /trunk/sharelib/pom.xml 1200045
> /trunk/webapp/pom.xml 1200045
>
> Diff: https://reviews.apache.org/r/2778/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by "Angelo K. Huang" <an...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/#review3308
-----------------------------------------------------------
I noticed that you have used hadoop.jars.scope at webapp. Why not use it at other places as well?
- Angelo K.
On 2011-11-10 00:06:30, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2778/
> -----------------------------------------------------------
>
> (Updated 2011-11-10 00:06:30)
>
>
> Review request for oozie, Tom White and Roman Shaposhnik.
>
>
> Summary
> -------
>
> Make Oozie compile against Hadoop 0.23.
>
> It depends on a few outstanding Hadoop JIRAs.
>
> Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
>
> This patch is a WIP
>
>
> This addresses bug OOZIE-565.
> https://issues.apache.org/jira/browse/OOZIE-565
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1200045
> /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1200045
> /trunk/examples/pom.xml 1200045
> /trunk/pom.xml 1200045
> /trunk/sharelib/pom.xml 1200045
> /trunk/webapp/pom.xml 1200045
>
> Diff: https://reviews.apache.org/r/2778/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Mohammad Islam <mi...@yahoo.com>.
> On 2011-11-22 03:44:59, Mohammad Islam wrote:
> > Are you working on anything in this regard? Looks like the issue is no in Hadoop/MR core.
> > We can commit it and create another one when needed.
> >
>
> Alejandro Abdelnur wrote:
> Yes, i'm working on zeroing the issue in Hadoop side.
>
> I'm for committing this and follow up if necessary once we identify the Hadoop issue.
Mona also faced the similar issue. doAs() is not working for Job submission. Hadoop uses the superuser (oozie) when submit MR job instead of the actual user. However, FileSystem looks ok.
Yes you can commit it and wait for the next hadoop patch. If oozie needs to be patched, we can create another JIRA.
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/#review3423
-----------------------------------------------------------
On 2011-11-21 21:02:32, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2778/
> -----------------------------------------------------------
>
> (Updated 2011-11-21 21:02:32)
>
>
> Review request for oozie, Tom White and Roman Shaposhnik.
>
>
> Summary
> -------
>
> Make Oozie compile against Hadoop 0.23.
>
> It depends on a few outstanding Hadoop JIRAs.
>
> Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
>
> This patch is a WIP
>
>
> This addresses bug OOZIE-565.
> https://issues.apache.org/jira/browse/OOZIE-565
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1204691
> /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1204691
> /trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 1204691
> /trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1204691
> /trunk/examples/pom.xml 1204691
> /trunk/pom.xml 1204691
> /trunk/sharelib/pom.xml 1204691
> /trunk/webapp/pom.xml 1204691
>
> Diff: https://reviews.apache.org/r/2778/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
> On 2011-11-22 03:44:59, Mohammad Islam wrote:
> > Are you working on anything in this regard? Looks like the issue is no in Hadoop/MR core.
> > We can commit it and create another one when needed.
> >
Yes, i'm working on zeroing the issue in Hadoop side.
I'm for committing this and follow up if necessary once we identify the Hadoop issue.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/#review3423
-----------------------------------------------------------
On 2011-11-21 21:02:32, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2778/
> -----------------------------------------------------------
>
> (Updated 2011-11-21 21:02:32)
>
>
> Review request for oozie, Tom White and Roman Shaposhnik.
>
>
> Summary
> -------
>
> Make Oozie compile against Hadoop 0.23.
>
> It depends on a few outstanding Hadoop JIRAs.
>
> Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
>
> This patch is a WIP
>
>
> This addresses bug OOZIE-565.
> https://issues.apache.org/jira/browse/OOZIE-565
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1204691
> /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1204691
> /trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 1204691
> /trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1204691
> /trunk/examples/pom.xml 1204691
> /trunk/pom.xml 1204691
> /trunk/sharelib/pom.xml 1204691
> /trunk/webapp/pom.xml 1204691
>
> Diff: https://reviews.apache.org/r/2778/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/#review3423
-----------------------------------------------------------
Are you working on anything in this regard? Looks like the issue is no in Hadoop/MR core.
We can commit it and create another one when needed.
- Mohammad
On 2011-11-21 21:02:32, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2778/
> -----------------------------------------------------------
>
> (Updated 2011-11-21 21:02:32)
>
>
> Review request for oozie, Tom White and Roman Shaposhnik.
>
>
> Summary
> -------
>
> Make Oozie compile against Hadoop 0.23.
>
> It depends on a few outstanding Hadoop JIRAs.
>
> Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
>
> This patch is a WIP
>
>
> This addresses bug OOZIE-565.
> https://issues.apache.org/jira/browse/OOZIE-565
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1204691
> /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1204691
> /trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 1204691
> /trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1204691
> /trunk/examples/pom.xml 1204691
> /trunk/pom.xml 1204691
> /trunk/sharelib/pom.xml 1204691
> /trunk/webapp/pom.xml 1204691
>
> Diff: https://reviews.apache.org/r/2778/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/
-----------------------------------------------------------
(Updated 2011-11-21 21:02:32.370561)
Review request for oozie, Tom White and Roman Shaposhnik.
Changes
-------
Patch rebased to trunk (after OOZIE-602 it was not applying).
It contains some minor (related) POM clean up (comments).
It changes 2 testcases to use the HadoopAccessor service (instead instantiating JobClient directly).
IMPORTANT: testcases are not passing as it seems proxyusers are not being honored by the new Mini MR.
Still, everything compiles.
As it is a non-default profile (0.23 is not used by default) I'd like to commit it and follow up with another JIRA in case the Mini MR issue requires tweaks in Oozie side.
Summary
-------
Make Oozie compile against Hadoop 0.23.
It depends on a few outstanding Hadoop JIRAs.
Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
This patch is a WIP
This addresses bug OOZIE-565.
https://issues.apache.org/jira/browse/OOZIE-565
Diffs (updated)
-----
/trunk/core/pom.xml 1204691
/trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1204691
/trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 1204691
/trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1204691
/trunk/examples/pom.xml 1204691
/trunk/pom.xml 1204691
/trunk/sharelib/pom.xml 1204691
/trunk/webapp/pom.xml 1204691
Diff: https://reviews.apache.org/r/2778/diff
Testing
-------
Thanks,
Alejandro
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
> On 2011-11-18 03:04:32, Mohammad Islam wrote:
> > /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java, line 132
> > <https://reviews.apache.org/r/2778/diff/2/?file=57266#file57266line132>
> >
> > Why it is required?
> > How will it work on MR1?
Yes, it works fine with pre 0.23
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/#review3341
-----------------------------------------------------------
On 2011-11-10 00:06:30, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2778/
> -----------------------------------------------------------
>
> (Updated 2011-11-10 00:06:30)
>
>
> Review request for oozie, Tom White and Roman Shaposhnik.
>
>
> Summary
> -------
>
> Make Oozie compile against Hadoop 0.23.
>
> It depends on a few outstanding Hadoop JIRAs.
>
> Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
>
> This patch is a WIP
>
>
> This addresses bug OOZIE-565.
> https://issues.apache.org/jira/browse/OOZIE-565
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1200045
> /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1200045
> /trunk/examples/pom.xml 1200045
> /trunk/pom.xml 1200045
> /trunk/sharelib/pom.xml 1200045
> /trunk/webapp/pom.xml 1200045
>
> Diff: https://reviews.apache.org/r/2778/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/#review3341
-----------------------------------------------------------
1. 'includeHadoopJars' is for including the hadoop jars in the tar ball. right?
If we don't want to inject it during build. we can still inject that through 'addtowar'script. right? That mean hadoop 0.23 will still supports the 'addtowar.sh' based jar injection.
2. So profile 'mr2' is for place holder. by default 'mr1' is activated.
/trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java
<https://reviews.apache.org/r/2778/#comment7481>
Why it is required?
How will it work on MR1?
- Mohammad
On 2011-11-10 00:06:30, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2778/
> -----------------------------------------------------------
>
> (Updated 2011-11-10 00:06:30)
>
>
> Review request for oozie, Tom White and Roman Shaposhnik.
>
>
> Summary
> -------
>
> Make Oozie compile against Hadoop 0.23.
>
> It depends on a few outstanding Hadoop JIRAs.
>
> Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
>
> This patch is a WIP
>
>
> This addresses bug OOZIE-565.
> https://issues.apache.org/jira/browse/OOZIE-565
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1200045
> /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1200045
> /trunk/examples/pom.xml 1200045
> /trunk/pom.xml 1200045
> /trunk/sharelib/pom.xml 1200045
> /trunk/webapp/pom.xml 1200045
>
> Diff: https://reviews.apache.org/r/2778/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/#review3371
-----------------------------------------------------------
Ship it!
+1
- Mohammad
On 2011-11-10 00:06:30, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2778/
> -----------------------------------------------------------
>
> (Updated 2011-11-10 00:06:30)
>
>
> Review request for oozie, Tom White and Roman Shaposhnik.
>
>
> Summary
> -------
>
> Make Oozie compile against Hadoop 0.23.
>
> It depends on a few outstanding Hadoop JIRAs.
>
> Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
>
> This patch is a WIP
>
>
> This addresses bug OOZIE-565.
> https://issues.apache.org/jira/browse/OOZIE-565
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1200045
> /trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1200045
> /trunk/examples/pom.xml 1200045
> /trunk/pom.xml 1200045
> /trunk/sharelib/pom.xml 1200045
> /trunk/webapp/pom.xml 1200045
>
> Diff: https://reviews.apache.org/r/2778/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: Make Oozie compile against Hadoop 0.23
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2778/
-----------------------------------------------------------
(Updated 2011-11-10 00:06:30.901190)
Review request for oozie, Tom White and Roman Shaposhnik.
Changes
-------
v2 of the patch:
* corrections in the dependencies inclusions/exclusions.
* added 2 profiles mr1 (default, for pre 0.23) and mr2 (for 0.23+)
* the includeHadoopJars profile works correctly with both mr1 and mr2 profiles
I'm able to build Oozie (cannot run testcases yet). When running Oozie against Hadoop 0.23.FOO Oozie is able to submit jobs to Hadoop but the actions are failing. Have to see what wrong there.
Hadoop.23.FOO = Hadoop 0.23.0 + MAPREDUCE-3370 + MAPREDUCE-3169 + HADOOP-7590 + MAPREDUCE-3380
Summary
-------
Make Oozie compile against Hadoop 0.23.
It depends on a few outstanding Hadoop JIRAs.
Everything compiles, testcases using MiniMR fail due to MAPREDUCE-3380.
This patch is a WIP
This addresses bug OOZIE-565.
https://issues.apache.org/jira/browse/OOZIE-565
Diffs (updated)
-----
/trunk/core/pom.xml 1200045
/trunk/core/src/main/java/org/apache/oozie/service/KerberosHadoopAccessorService.java 1200045
/trunk/examples/pom.xml 1200045
/trunk/pom.xml 1200045
/trunk/sharelib/pom.xml 1200045
/trunk/webapp/pom.xml 1200045
Diff: https://reviews.apache.org/r/2778/diff
Testing
-------
Thanks,
Alejandro