[jira] [Updated] (PIG-2784) Framework for dynamic query optimization

["Aniket Mokashi (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aniket Mokashi updated PIG-2784:
--------------------------------

    Labels: GSOC2014  (was: )

> Framework for dynamic query optimization
> ----------------------------------------
>
>                 Key: PIG-2784
>                 URL: https://issues.apache.org/jira/browse/PIG-2784
>             Project: Pig
>          Issue Type: New Feature
>            Reporter: Jie Li
>            Assignee: Aniket Mokashi
>              Labels: GSOC2014
>
> We need a framework to implement dynamic query optimization, i.e. changing the query plan at runtime. Currently we support estimating the number of reducers dynamically, which works well as the first step but was not perfectly implemented. In near future, we'll support more dynamic optimization, like [removing sample job for order-by|https://issues.apache.org/jira/browse/PIG-483], [removing limit job|https://issues.apache.org/jira/browse/PIG-2675], dynamically detecting skew and using skew-join, etc.
> Currently estimating #reducer is implemented in JobControlCompiler after MRCompiler compiles all the MapReduceOperators and generate the complete MRPlan. One place (discussed with Thejas) to implement the framework is at the MRCompiler, where the MRPlan'll be generated at batches and adjusted dynamically. 
> Any comment?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)