You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by "Duckworth, Will" <wd...@comscore.com> on 2012/10/10 18:11:21 UTC
Long time registering python UDF
Wanted to see if anyone else is seeing this behavior.
I have a python file with a single 40 line function/UDF that seems to take 20+ minutes to get registered. I don’t see the same issue when I register a Java UDF. Is something that I am doing (or not doing)?
2012-10-10 10:36:33,862 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine - Register scripting UDF: udf.explode
2012-10-10 10:49:29,018 [main] INFO org.apache.pig.scripting.jython.JythonFunction - Schema 'R:bag{T:tuple( ... )}' defined for func explode
2012-10-10 11:00:45,036 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: HASH_JOIN
,GROUP_BY,FILTER
Thanks.
Will Duckworth Senior Vice President, Software Engineering | comScore, Inc. (NASDAQ:SCOR)
o +1 (703) 438-2108 | m +1 (301) 606-2977 | wduckworth@comscore.com<ma...@comscore.com>
...........................................................................................................
Introducing Mobile Metrix 2.0 - The next generation of mobile behavioral measurement
www.comscore.com/MobileMetrix<http://www.comscore.com/Products_Services/Product_Index/Mobile_Metrix_2.0>
RE: Long time registering python UDF
Posted by "Duckworth, Will" <wd...@comscore.com>.
A single import:
from org.apache.pig.tools.counters import PigCounterHelper
Will Duckworth Senior Vice President, Software Engineering | comScore, Inc.(NASDAQ:SCOR)
o +1 (703) 438-2108 | m +1 (301) 606-2977 | mailto:wduckworth@comscore.com
.....................................................................................................
-----Original Message-----
From: Russell Jurney [mailto:russell.jurney@gmail.com]
Sent: Wednesday, October 10, 2012 2:10 PM
To: user@pig.apache.org
Subject: Re: Long time registering python UDF
Curious: does the UDF have a lot of includes?
Russell Jurney twitter.com/rjurney
On Oct 10, 2012, at 9:11 AM, "Duckworth, Will" <wd...@comscore.com> wrote:
> Wanted to see if anyone else is seeing this behavior.
>
> I have a python file with a single 40 line function/UDF that seems to take 20+ minutes to get registered. I don't see the same issue when I register a Java UDF. Is something that I am doing (or not doing)?
>
> 2012-10-10 10:36:33,862 [main] INFO
> org.apache.pig.scripting.jython.JythonScriptEngine - Register
> scripting UDF: udf.explode
> 2012-10-10 10:49:29,018 [main] INFO
> org.apache.pig.scripting.jython.JythonFunction - Schema
> 'R:bag{T:tuple( ... )}' defined for func explode
> 2012-10-10 11:00:45,036 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: HASH_JOIN ,GROUP_BY,FILTER
>
> Thanks.
>
>
>
> Will Duckworth Senior Vice President, Software Engineering | comScore,
> Inc. (NASDAQ:SCOR)
>
> o +1 (703) 438-2108 | m +1 (301) 606-2977 |
> wduckworth@comscore.com<ma...@comscore.com>
>
> ...........................................................................................................
>
> Introducing Mobile Metrix 2.0 - The next generation of mobile
> behavioral measurement
> www.comscore.com/MobileMetrix<http://www.comscore.com/Products_Service
> s/Product_Index/Mobile_Metrix_2.0>
>
>
Re: Long time registering python UDF
Posted by Russell Jurney <ru...@gmail.com>.
Curious: does the UDF have a lot of includes?
Russell Jurney twitter.com/rjurney
On Oct 10, 2012, at 9:11 AM, "Duckworth, Will" <wd...@comscore.com> wrote:
> Wanted to see if anyone else is seeing this behavior.
>
> I have a python file with a single 40 line function/UDF that seems to take 20+ minutes to get registered. I don’t see the same issue when I register a Java UDF. Is something that I am doing (or not doing)?
>
> 2012-10-10 10:36:33,862 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine - Register scripting UDF: udf.explode
> 2012-10-10 10:49:29,018 [main] INFO org.apache.pig.scripting.jython.JythonFunction - Schema 'R:bag{T:tuple( ... )}' defined for func explode
> 2012-10-10 11:00:45,036 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: HASH_JOIN
> ,GROUP_BY,FILTER
>
> Thanks.
>
>
>
> Will Duckworth Senior Vice President, Software Engineering | comScore, Inc. (NASDAQ:SCOR)
>
> o +1 (703) 438-2108 | m +1 (301) 606-2977 | wduckworth@comscore.com<ma...@comscore.com>
>
> ...........................................................................................................
>
> Introducing Mobile Metrix 2.0 - The next generation of mobile behavioral measurement
> www.comscore.com/MobileMetrix<http://www.comscore.com/Products_Services/Product_Index/Mobile_Metrix_2.0>
>
>