You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Xiaomeng Wan <sh...@gmail.com> on 2011/01/25 00:54:10 UTC
python udf doesnt work
Hi,
I want to write a python udf to split string into bags
------------------------------------------------------------
#!/usr/bin/python
import re
@outputSchema("y:bag{t:tuple(word:chararray)}")
def strsplittobag(content,regex):
return re.compile(regex).split(content)
------------------------------------------------------------
it gave an error saying "could not instantiate
'org.apache.pig.scripting.jython.JythonFunction' with arguments
'[/home/.../mypyudfs.py, strsplittobag]'". I had some other python
udfs working, so shouldn't be configuration problem. I am new to
python, did I miss anything?
Thanks!
Shawn
Re: python udf doesnt work
Posted by Xiaomeng Wan <sh...@gmail.com>.
It works! Only need to explicitly cast the results into bag of tuples.
from java.util.regex import *
from java.lang import *
@outputSchema("y:bag{t:tuple(word:chararray)}")
def strsplittobag(content,regex):
toks = Pattern.compile(regex).split(content)
outBag = []
for tok in toks:
tup = tok,
outBag.append(tup)
return outBag
Thank you all!
Shawn
On Wed, Jan 26, 2011 at 11:01 AM, Julien Le Dem <le...@yahoo-inc.com> wrote:
> As a workaround, in Jython you can also use the java classes.
> Something like: (not tested)
>
> from java.util.regex import *
> from java.lang import *
>
> @outputSchema("y:bag{t:tuple(word:chararray)}")
> def strsplittobag(content,regex):
> return Pattern.compile(regex).split(content)
>
> Julien
>
> On 1/25/11 5:46 PM, "Richard Ding" <rd...@yahoo-inc.com> wrote:
>
> You're right. There're two issues here. First, the Jython script needs to
> locate the modules in its search path (e.g. python.path). If you have the
> right env variable set, Jython script should be able to find and import the
> module. Second, Pig currently doesn't automatically ship the module file to
> the backend, so even if you set the search path in the frontend, the backend
> still cannot locate the module.
>
> Finally, there is incompatibility between Python modules and Jython modules.
> You need to use Jython modules that come with Jython installation (in the
> Lib directory).
>
> We're looking into these issues and hoping to provide a solution in the next
> release.
>
> Thanks,
> -Richard
>
>
> On 1/25/11 12:50 PM, "Xiaomeng Wan" <sh...@gmail.com> wrote:
>
> Hi Daniel,
>
> I did put jython.jar in classpath. By comparing other python udfs with
> this one, I find those udfs which work do not import anything. Could
> that be the cause? Do I need to anything extra to import module in my
> udf?
>
> Thanks!
>
> Shawn
>
> On Mon, Jan 24, 2011 at 5:28 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:
>> Put build/ivy/lib/Pig/jython-2.5.0.jar in your classpath (if not there, do
>> ant first). This is a bug we need to fix.
>>
>> Daniel
>>
>> Xiaomeng Wan wrote:
>>>
>>> Hi,
>>> I want to write a python udf to split string into bags
>>>
>>> ------------------------------------------------------------
>>> #!/usr/bin/python
>>>
>>> import re
>>> @outputSchema("y:bag{t:tuple(word:chararray)}")
>>> def strsplittobag(content,regex):
>>> return re.compile(regex).split(content)
>>> ------------------------------------------------------------
>>>
>>> it gave an error saying "could not instantiate
>>> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
>>> '[/home/.../mypyudfs.py, strsplittobag]'". I had some other python
>>> udfs working, so shouldn't be configuration problem. I am new to
>>> python, did I miss anything?
>>>
>>> Thanks!
>>>
>>> Shawn
>>>
>>
>>
>
>
>
Re: python udf doesnt work
Posted by Julien Le Dem <le...@yahoo-inc.com>.
As a workaround, in Jython you can also use the java classes.
Something like: (not tested)
from java.util.regex import *
from java.lang import *
@outputSchema("y:bag{t:tuple(word:chararray)}")
def strsplittobag(content,regex):
return Pattern.compile(regex).split(content)
Julien
On 1/25/11 5:46 PM, "Richard Ding" <rd...@yahoo-inc.com> wrote:
You're right. There're two issues here. First, the Jython script needs to locate the modules in its search path (e.g. python.path). If you have the right env variable set, Jython script should be able to find and import the module. Second, Pig currently doesn't automatically ship the module file to the backend, so even if you set the search path in the frontend, the backend still cannot locate the module.
Finally, there is incompatibility between Python modules and Jython modules. You need to use Jython modules that come with Jython installation (in the Lib directory).
We're looking into these issues and hoping to provide a solution in the next release.
Thanks,
-Richard
On 1/25/11 12:50 PM, "Xiaomeng Wan" <sh...@gmail.com> wrote:
Hi Daniel,
I did put jython.jar in classpath. By comparing other python udfs with
this one, I find those udfs which work do not import anything. Could
that be the cause? Do I need to anything extra to import module in my
udf?
Thanks!
Shawn
On Mon, Jan 24, 2011 at 5:28 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:
> Put build/ivy/lib/Pig/jython-2.5.0.jar in your classpath (if not there, do
> ant first). This is a bug we need to fix.
>
> Daniel
>
> Xiaomeng Wan wrote:
>>
>> Hi,
>> I want to write a python udf to split string into bags
>>
>> ------------------------------------------------------------
>> #!/usr/bin/python
>>
>> import re
>> @outputSchema("y:bag{t:tuple(word:chararray)}")
>> def strsplittobag(content,regex):
>> return re.compile(regex).split(content)
>> ------------------------------------------------------------
>>
>> it gave an error saying "could not instantiate
>> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
>> '[/home/.../mypyudfs.py, strsplittobag]'". I had some other python
>> udfs working, so shouldn't be configuration problem. I am new to
>> python, did I miss anything?
>>
>> Thanks!
>>
>> Shawn
>>
>
>
Re: python udf doesnt work
Posted by Richard Ding <rd...@yahoo-inc.com>.
You're right. There're two issues here. First, the Jython script needs to locate the modules in its search path (e.g. python.path). If you have the right env variable set, Jython script should be able to find and import the module. Second, Pig currently doesn't automatically ship the module file to the backend, so even if you set the search path in the frontend, the backend still cannot locate the module.
Finally, there is incompatibility between Python modules and Jython modules. You need to use Jython modules that come with Jython installation (in the Lib directory).
We're looking into these issues and hoping to provide a solution in the next release.
Thanks,
-Richard
On 1/25/11 12:50 PM, "Xiaomeng Wan" <sh...@gmail.com> wrote:
Hi Daniel,
I did put jython.jar in classpath. By comparing other python udfs with
this one, I find those udfs which work do not import anything. Could
that be the cause? Do I need to anything extra to import module in my
udf?
Thanks!
Shawn
On Mon, Jan 24, 2011 at 5:28 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:
> Put build/ivy/lib/Pig/jython-2.5.0.jar in your classpath (if not there, do
> ant first). This is a bug we need to fix.
>
> Daniel
>
> Xiaomeng Wan wrote:
>>
>> Hi,
>> I want to write a python udf to split string into bags
>>
>> ------------------------------------------------------------
>> #!/usr/bin/python
>>
>> import re
>> @outputSchema("y:bag{t:tuple(word:chararray)}")
>> def strsplittobag(content,regex):
>> return re.compile(regex).split(content)
>> ------------------------------------------------------------
>>
>> it gave an error saying "could not instantiate
>> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
>> '[/home/.../mypyudfs.py, strsplittobag]'". I had some other python
>> udfs working, so shouldn't be configuration problem. I am new to
>> python, did I miss anything?
>>
>> Thanks!
>>
>> Shawn
>>
>
>
Re: python udf doesnt work
Posted by Xiaomeng Wan <sh...@gmail.com>.
Hi Daniel,
I did put jython.jar in classpath. By comparing other python udfs with
this one, I find those udfs which work do not import anything. Could
that be the cause? Do I need to anything extra to import module in my
udf?
Thanks!
Shawn
On Mon, Jan 24, 2011 at 5:28 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:
> Put build/ivy/lib/Pig/jython-2.5.0.jar in your classpath (if not there, do
> ant first). This is a bug we need to fix.
>
> Daniel
>
> Xiaomeng Wan wrote:
>>
>> Hi,
>> I want to write a python udf to split string into bags
>>
>> ------------------------------------------------------------
>> #!/usr/bin/python
>>
>> import re
>> @outputSchema("y:bag{t:tuple(word:chararray)}")
>> def strsplittobag(content,regex):
>> return re.compile(regex).split(content)
>> ------------------------------------------------------------
>>
>> it gave an error saying "could not instantiate
>> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
>> '[/home/.../mypyudfs.py, strsplittobag]'". I had some other python
>> udfs working, so shouldn't be configuration problem. I am new to
>> python, did I miss anything?
>>
>> Thanks!
>>
>> Shawn
>>
>
>
Re: python udf doesnt work
Posted by Daniel Dai <ji...@yahoo-inc.com>.
Put build/ivy/lib/Pig/jython-2.5.0.jar in your classpath (if not there,
do ant first). This is a bug we need to fix.
Daniel
Xiaomeng Wan wrote:
> Hi,
> I want to write a python udf to split string into bags
>
> ------------------------------------------------------------
> #!/usr/bin/python
>
> import re
> @outputSchema("y:bag{t:tuple(word:chararray)}")
> def strsplittobag(content,regex):
> return re.compile(regex).split(content)
> ------------------------------------------------------------
>
> it gave an error saying "could not instantiate
> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
> '[/home/.../mypyudfs.py, strsplittobag]'". I had some other python
> udfs working, so shouldn't be configuration problem. I am new to
> python, did I miss anything?
>
> Thanks!
>
> Shawn
>