You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rajitha (JIRA)" <ji...@apache.org> on 2014/02/24 05:59:20 UTC
[jira] [Commented] (PIG-2784) Framework for dynamic query
optimization
[ https://issues.apache.org/jira/browse/PIG-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13910021#comment-13910021 ]
Rajitha commented on PIG-2784:
------------------------------
I am very interested in contributing this project in GSoC 2014.
I'm Rajitha Ranasinghe, third year undergraduate at the University of Moratuwa,Sri Lanka. As I'm currently working as a software engineering trainee in an open source company, I am involved in several open source projects and have a good practical knowledge and experience in open source project development.
Currently I'm referring to the learning materials you have provided regarding the project to gain background knowledge. I need to discuss about this project. How can I proceed?
> Framework for dynamic query optimization
> ----------------------------------------
>
> Key: PIG-2784
> URL: https://issues.apache.org/jira/browse/PIG-2784
> Project: Pig
> Issue Type: New Feature
> Reporter: Jie Li
> Labels: GSOC2014
>
> We need a framework to implement dynamic query optimization, i.e. changing the query plan at runtime. Currently we support estimating the number of reducers dynamically, which works well as the first step but was not perfectly implemented. In near future, we'll support more dynamic optimization, like [removing sample job for order-by|https://issues.apache.org/jira/browse/PIG-483], [removing limit job|https://issues.apache.org/jira/browse/PIG-2675], dynamically detecting skew and using skew-join, etc.
> Currently estimating #reducer is implemented in JobControlCompiler after MRCompiler compiles all the MapReduceOperators and generate the complete MRPlan. One place (discussed with Thejas) to implement the framework is at the MRCompiler, where the MRPlan'll be generated at batches and adjusted dynamically.
> Any comment?
> This is a candidate project for Google summer of code 2014. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2014
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)