You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2017/02/01 15:51:51 UTC
[jira] [Commented] (PIG-4913) Reduce jython function initiation
during compilation
[ https://issues.apache.org/jira/browse/PIG-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848538#comment-15848538 ]
Rohini Palaniswamy commented on PIG-4913:
-----------------------------------------
My concern was same script was being recompiled multiple times. Reading from the file multiple times is just a side effect of that. The fix should be that we don't recompile again.
> Reduce jython function initiation during compilation
> ----------------------------------------------------
>
> Key: PIG-4913
> URL: https://issues.apache.org/jira/browse/PIG-4913
> Project: Pig
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
> Assignee: Adam Szita
> Attachments: PIG-4913.patch
>
>
> While investigating PIG-4908, saw that ScriptEngine.getScriptAsStream was invoked way too many times during compilation phase for a simple script.
> {code:title=sleep.py}
> #!/usr/bin/python
> import time;
> @outputSchema("sltime:int")
> def sleep(num):
> if num == 1:
> print "Sleeping for %d minutes" % num;
> time.sleep(num * 60);
> return num;
> {code}
> {code:title=sleep.pig}
> register 'sleep.py' using jython;
> A = LOAD '/tmp/sleepdata' as (f1:int);
> B = FOREACH A generate $0, sleep($0);
> STORE B into '/tmp/tezout';
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)