You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oozie.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2011/09/08 07:15:09 UTC
[jira] [Created] (OOZIE-308) GH-480: Support new Pig API to submit
pig job
GH-480: Support new Pig API to submit pig job
---------------------------------------------
Key: OOZIE-308
URL: https://issues.apache.org/jira/browse/OOZIE-308
Project: Oozie
Issue Type: Bug
Reporter: Hadoop QA
In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-308) GH-480: Support new Pig API to
submit pig job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101913#comment-13101913 ]
Hadoop QA commented on OOZIE-308:
---------------------------------
brookwc remarked:
Yes, we are considering to just add a small piece of reflection logic in there to decide which class to use. Seems we do NOT need to introduce any new configuration variable. However, we do plan to create a new PigMain - PigMainNew (a better class name is needed) class for this. The old PigMain will be kept intact - users can force to use it by specifying it as <MainClass> in pig action xml definition.
Chao
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-308) GH-480: Support new Pig API to
submit pig job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101912#comment-13101912 ]
Hadoop QA commented on OOZIE-308:
---------------------------------
anew remarked:
That would mean yet another configuration that has to be provided?
is wrong about using inflection to determine which pig version is in use? The change is isolated to the method PigMain.runPigJob(). I don't see harm in adding some logic there that decides which class to use.
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-308) GH-480: Support new Pig API to
submit pig job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101916#comment-13101916 ]
Hadoop QA commented on OOZIE-308:
---------------------------------
tucu00 remarked:
If the current PigMain class is modified to use the new Pig API then Oozie will stop compiling when using an older versions of Pig and it may not even run with an oder version of Pig.
The patch should accommodate for compilation/execution with older versions of Pig
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-308) GH-480: Support new Pig API to
submit pig job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099980#comment-13099980 ]
Hadoop QA commented on OOZIE-308:
---------------------------------
brookwc remarked:
Pig team suggests Oozie to use the new API PigRunner.run() to submit and run Pig jobs. They do NOT plan to support use of Main.main(args) in Oozie in future. As such, we plan to adopt this new API. Also for backward-compatibility concern, we need to support old Pig version as well.
Our proposal is as follows:
Oozie will check if new class/API is available on classpath through java reflection, if it is (Pig-0.8+), Oozie will use it. Otherwise Oozie will still use Main.main().
Your comments and thoughts are welcome.
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-308) GH-480: Support new Pig API to
submit pig job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101914#comment-13101914 ]
Hadoop QA commented on OOZIE-308:
---------------------------------
anew remarked:
Why keep the old class and duplicate code? Are you concerned that users may depend on use of pig.Main because PigRunner will not provide the same options? Afaik, there will be no loss of functionality with PigRunner.
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-308) GH-480: Support new Pig API to
submit pig job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101911#comment-13101911 ]
Hadoop QA commented on OOZIE-308:
---------------------------------
angelokh remarked:
We actually plan to create a new PigMain like you have just suggested, and also add a new configuration to control what's default.
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-308) GH-480: Support new Pig API to
submit pig job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101915#comment-13101915 ]
Hadoop QA commented on OOZIE-308:
---------------------------------
mislam77 remarked:
The proposal is to modify the current PigMain to accommodate PigRunner and old Main.amin() at the same time.
However, we want to keep the old pig main temporarily in different name. The reason is operational.
Pig is used extensively in the production. PigRunner is not used by anybody yet. Oozie will be the first user. That's why, we are expecting some issues that might slip our Oozie QE too.
The plan is: if that happens for any use case, we could ask that user to use the old main by adding a new configuration in workflow.xml. That way, the user will be unblocked for the time being. At the same time, we would work with pig team to resolve the root cause of using PigRunner.
This is just for sometime until we fell comfortable with the new API. After that we will throw away that undocumented code.
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (OOZIE-308) GH-480: Support new Pig API to submit
pig job
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik closed OOZIE-308.
----------------------------------
Resolution: Fixed
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-308) GH-480: Support new Pig API to
submit pig job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101917#comment-13101917 ]
Hadoop QA commented on OOZIE-308:
---------------------------------
brookwc remarked:
I already created a PR for this. What I did is to download pig-0.8.0 artifact from maven repo for compilation.
Pig team already made the artifact available there.
http://mail-archives.apache.org/mod_mbox/pig-user/201101.mbox/%3CAANLkTi=bix2i4AzegTjXAJXfigycOHgvdPq=kmpOb_zf@mail.gmail.com%3E
Using 0.8.0 artifact, oozie compiles fine. I also tested run time with different pig versions, ie 0.7 and 0.8, it works fine.
But with your comment on "Pig POM includes all dependencies with 'compile' scope", we need to take a closer look into it.
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-308) GH-480: Support new Pig API to
submit pig job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101918#comment-13101918 ]
Hadoop QA commented on OOZIE-308:
---------------------------------
mislam77 remarked:
Closed by b10bc196639405dd4a9cd1ca434c49d0c2044123 Support new Pig API to submit pig job
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-308) GH-480: Support new Pig API to
submit pig job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101910#comment-13101910 ]
Hadoop QA commented on OOZIE-308:
---------------------------------
tucu00 remarked:
How about having an new Oozie PigMain class that uses Pig new API, and PigActionExecutor would read from Oozie configuration which is the default PigMain class to use?
> GH-480: Support new Pig API to submit pig job
> ---------------------------------------------
>
> Key: OOZIE-308
> URL: https://issues.apache.org/jira/browse/OOZIE-308
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> In addition of supporting old pig release, oozie needs to support PigRunner API for pig 0.8+.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira