You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by "Nick White (JIRA)" <ji...@apache.org> on 2012/12/13 05:25:22 UTC

[jira] [Updated] (PIG-3095) "which" is called many, many times for each Pig STREAM statement

     [ https://issues.apache.org/jira/browse/PIG-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nick White updated PIG-3095:
----------------------------

    Attachment: PIG-3095.patch
    
> "which" is called many, many times for each Pig STREAM statement
> ----------------------------------------------------------------
>
>                 Key: PIG-3095
>                 URL: https://issues.apache.org/jira/browse/PIG-3095
>             Project: Pig
>          Issue Type: Bug
>          Components: grunt, impl
>    Affects Versions: 0.12
>            Reporter: Nick White
>            Assignee: Nick White
>              Labels: patch, performance
>             Fix For: 0.12
>
>         Attachments: PIG-3095.patch
>
>
> STREAM statements are checked by the LogicalPlanBuilder as it comes across them - and these checks include running the system utility "which". However, due to the backtracking parsing mechanism "which" is called repeatedly with the same arguments (I noticed this while profiling a script with 4 STREAM statements - "which" was run over 230 times!). The attached patch just caches the return value of "which", reducing the overhead of running a system process to a Map lookup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira