You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Julien Le Dem (JIRA)" <ji...@apache.org> on 2010/07/01 22:31:52 UTC
[jira] Created: (PIG-1479) Embed Pig in scripting languages
Embed Pig in scripting languages
--------------------------------
Key: PIG-1479
URL: https://issues.apache.org/jira/browse/PIG-1479
Project: Pig
Issue Type: New Feature
Reporter: Julien Le Dem
It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1479) Embed Pig in scripting languages
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1479:
------------------------------
Attachment: PIG-1479_2.patch
In the previous patch, the executeScript method on ScriptPigServer returns a list of ExecJobs (one for each store statement in the script). Unfortunately, the order of ExecJobs in the list is indeterminate.
This patch fixes this problem by making the executeScript method return a PigStats object. One then can retrieves the output result by the alias corresponding to store statement.
Here is a example:
{code}
P = pig.executeScript("""
A = load '${input}';
... ...
store G into '${output}'; """)
output = P.result("G") # an OutputStats object
iter = output.iterator()
if iter.hasNext():
# do something
else:
# do something else
{code}
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Attachments: PIG-1479.patch, PIG-1479_2.patch, pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1479) Embed Pig in scripting languages
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1479:
----------------------------
Assignee: Richard Ding
Fix Version/s: 0.9.0
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Assignee: Richard Ding
> Fix For: 0.9.0
>
> Attachments: PIG-1479.patch, PIG-1479_2.patch, pig-greek-test.tar, pig-greek-test.tar, pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1479) Embed Pig in scripting languages
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12909345#action_12909345 ]
Julien Le Dem commented on PIG-1479:
------------------------------------
Thanks Richard!
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Attachments: PIG-1479.patch, pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1479) Embed Pig in scripting languages
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913859#action_12913859 ]
Julien Le Dem commented on PIG-1479:
------------------------------------
Using the file extension requires a registration mechanism (or hard coded list) so if it is supported it would be nice to be able to provide the class name of the scripting implementation as well.
I would like to use my own implementation of the scripting engine (let's say javascript) by specifying the class name in the command line.
similar to the mecanism for UDFs inclusion:
http://wiki.apache.org/pig/UDFsUsingScriptingLanguages
{quote}
Register 'test.py' using org.apache.pig.scripting.jython.JythonScriptEngine as myfuncs;
{quote}
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Assignee: Richard Ding
> Fix For: 0.9.0
>
> Attachments: PIG-1479.patch, PIG-1479_2.patch, pig-greek-test.tar, pig-greek-test.tar, pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1479) Embed Pig in scripting languages
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928400#action_12928400 ]
Richard Ding commented on PIG-1479:
-----------------------------------
Alan has posted a proposal that includes embedding Pig in scripting language on Pig wiki: http://wiki.apache.org/pig/TuringCompletePig. The proposal is based on the implementation here via a JDBC like compile, bind, run model.
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Assignee: Richard Ding
> Fix For: 0.9.0
>
> Attachments: PIG-1479.patch, PIG-1479_2.patch, pig-greek-test.tar, pig-greek-test.tar, pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1479) Embed Pig in scripting languages
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1479:
------------------------------
Attachment: pig-greek-test.tar
Attach the test script modified based on Julien's comment. As for commend line option -g, it can also use one parameter (script file name) and let Pig determine the script engine by the file extension.
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Attachments: PIG-1479.patch, PIG-1479_2.patch, pig-greek-test.tar, pig-greek-test.tar, pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1479) Embed Pig in scripting languages
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem updated PIG-1479:
-------------------------------
Attachment: pig-greek.tgz
See: https://issues.apache.org/jira/browse/PIG-928
To run the example (assuming javac, jar and java are in your PATH):
* tar xzvf pig-greek.tgz
* add pig-0.6.0-core.jar to the lib folder
* ./makejar.sh
* ./runme.sh
This contains a generic base class and a Python implementation.
To implement other scripting languages, extend org.apache.pig.greek.ScriptEngine
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Attachments: pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1479) Embed Pig in scripting languages
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12909898#action_12909898 ]
Julien Le Dem commented on PIG-1479:
------------------------------------
The -g parameter on the command line should take two parameters, the scripting implementation instance name and the script itself.
That way we can have several scripting implementations.
{noformat}
java -cp pig.jar:<jython jar> org.apache.pig.Main -x local -g jython script/tc.py
{noformat}
{code}
case GREEK: {
ScriptEngine scriptEngine = ScriptEngine.getInstance(instanceName);
scriptEngine.run(new PigServer(pigContext), file);
return ReturnCode.SUCCESS;
}
{code}
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Attachments: PIG-1479.patch, PIG-1479_2.patch, pig-greek-test.tar, pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1479) Embed Pig in scripting languages
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1479:
------------------------------
Attachment: PIG-1479.patch
Thanks Julien. I rebased the patch with the latest trunk and added an option (-greek) in the Main class.
Now one can run a "PIG-Greek" script with following command:
{code}
java -cp pig.jar:<jython jar>:<hadoop config dir> org.apache.pig.Main -g <pig-greek script>
{code}
or in local mode:
{code}
java -cp pig.jar:<jython jar> org.apache.pig.Main -x local -g <pig-greek script>
{code}
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Attachments: PIG-1479.patch, pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1479) Embed Pig in scripting languages
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1479:
------------------------------
Attachment: pig-greek-test.tar
Attach the updated test program from Julien.
To run the example:
* tar -xvf pig-greek-test.tar
* java -cp pig.jar:<jython jar> org.apache.pig.Main -x local -g script/tc.py
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Attachments: PIG-1479.patch, PIG-1479_2.patch, pig-greek-test.tar, pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1479) Embed Pig in scripting languages
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12909901#action_12909901 ]
Julien Le Dem commented on PIG-1479:
------------------------------------
The end of loop condition in the script can just test for to_join_n emptiness. It was testing both because it did not know which one was to_join_n.
{code}
if (not P.result("to_join_n").iterator().hasNext()):
{code}
> Embed Pig in scripting languages
> --------------------------------
>
> Key: PIG-1479
> URL: https://issues.apache.org/jira/browse/PIG-1479
> Project: Pig
> Issue Type: New Feature
> Reporter: Julien Le Dem
> Attachments: PIG-1479.patch, PIG-1479_2.patch, pig-greek-test.tar, pig-greek.tgz
>
>
> It should be possible to embed Pig calls in a scripting language and let functions defined in the same script available as UDFs.
> This is a spin off of https://issues.apache.org/jira/browse/PIG-928 which lets users define UDFs in scripting languages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.