You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Alan Gates (JIRA)" <ji...@apache.org> on 2010/01/06 00:49:56 UTC
[jira] Created: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
LogicalPlan and Optimizer are too complex and hard to work with
---------------------------------------------------------------
Key: PIG-1178
URL: https://issues.apache.org/jira/browse/PIG-1178
Project: Pig
Issue Type: Improvement
Reporter: Alan Gates
The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Modi updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
I found a bug in the code so I'll be releasing another patch for the same.
I'll keep this patch in the JIRA until I replace it with a new one so everyone can review it.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: PIG-1178-6.patch
PIG-1178-6.patch is mostly for error handling
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: (was: PIG-1178-6.patch)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ying He updated PIG-1178:
-------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895463#action_12895463 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12451203/PIG-1178-5.patch
against trunk revision 982423.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 91 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/375/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803856#action_12803856 ]
Ying He commented on PIG-1178:
------------------------------
Alan,thanks for the review.
for 6), the predecessor of LOFilter would be LOJoin, so all projections would have input number 0. My algorithm is to get field names from column number. The field names after join would be like A::id, B::id, And findCommon() is to search for the longest prefix of these fields, to push filter to be after that alias. For example, if field names are A::id, and A::value, the filter is pushed after A. if field names are D::A::id and D::A::value, the filter can be pushed after D, then pushed further to be after A.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ying He updated PIG-1178:
-------------------------
Attachment: PIG_1178.patch
add test cases to TestExperimentalRule, and fix findbugs problems
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800353#action_12800353 ]
Ying He commented on PIG-1178:
------------------------------
to answer Daniel's questions:
. In Rule.match, is PatternMatchOperatorPlan only contains leave nodes but not edge information? If so, instead of saying "A list of all matched sub-plans", can we put more details in the comments?
The returned lists are plans. You can call getPredecessors() or getSuccessors() on any node in the plan. The implementation doesn't keep edge information, it calls the base plan for this information and return the operators that are in this sub-plan. So looking from outside, it is a plan, it's just read-only, and method to update the plan would throw an exception.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798969#action_12798969 ]
Daniel Dai commented on PIG-1178:
---------------------------------
Two minor observations:
1. PatternMatchOperatorPlan.parent does not sound very meanful, shall we change the name to basePlan?
2. In Rule.match, is PatternMatchOperatorPlan only contains leave nodes but not edge information? If so, instead of saying "A list of all matched sub-plans", can we put more details in the comments?
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845083#action_12845083 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12438738/pig_1178_3.3.patch
against trunk revision 922664.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 28 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/251/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/251/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/251/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892658#action_12892658 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12450250/PIG-1178-4.patch
against trunk revision 979362.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 48 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 446 release audit warnings (more than the trunk's current 398 warnings).
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-10.patch, PIG-1178-11.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, PIG-1178-8.patch, PIG-1178-9.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800054#action_12800054 ]
Alan Gates commented on PIG-1178:
---------------------------------
bq. in ProjectExpression, is it better to change the object variable "input" from "int" to "LogicalRelationalOperator" to point to the operator that the project expression operates on directly?
I want to avoid references to the actual relational operators in the expressions because it makes patching up the plans after a transformation much easier. If each project keeps a reference to the relational operator, then when the plan is transformed we have to go to every project and change its reference. By keeping pointers only to which input number, we don't have to make any changes in the projects after a transformation in the plan.
bq. And I don't understand why this operator needs alias it references. But if we change input to operator object, the alias can be get from the operator.
You're right, we shouldn't double store aliases here. We should just use the uid and the project reference. I'll make the change.
bq. I don't know the purpose of ColumnExpression. Is it to capture operands? It doesn't seem to have any special features. So I am not sure if it is necessary
It's a super class for all expressions that represent a single value: projection, constants, and eventually, tuple and map dereferences. I think it's useful for understanding the categorization of expressions. I'm not sure it adds any functionality.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: PIG-1178-11.patch
PIG-1178-11.patch change the layout of explain, error code and comments, etc. No real functional changes.
test-patch result:
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 11 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-10.patch, PIG-1178-11.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, PIG-1178-8.patch, PIG-1178-9.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: PIG-1178-5.patch
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800289#action_12800289 ]
Ying He commented on PIG-1178:
------------------------------
+1
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829759#action_12829759 ]
Alan Gates commented on PIG-1178:
---------------------------------
Comments that came out of a review of the twiki doc the pig team did:
1) In OperatorPlan, the use of roots and leaves in the graph was considered confusing. Some people view roots as sources and some as sinks. It was recommended that we switch roots to sources and leaves to sinks to avoid confusion.
2) The new OperatorPlan does not include mergeSharedPlan, which was used by multi-query functionality in the old plan. After further investigation I found that merge is currently only used by multi-query for physical plans. While ideally we would like to use this infrastructure for physical plans too, I feel it is reasonable to put off adding merge until at least the initial prototyping phase is done. After briefling looking at it I see no reason why it should not work, though we may need a more precise way to decide when two nodes are the same and should be merged.
3) A point was raised that perhaps the optimizer should reset the annotations on the nodes after a transform and all the attached listeners have been run. With further thought, I don't think so, as there may be annotations we want to last across transforms. For example, a rule that could match an infinite number of times may want to "sign" a node to note it's already been there so that it does not fire on the node again. The easiest way to do this signing would be with the annotations. However, I can see that there would be a desire to clear certain annotations so that each pass of the optimizer has a fresh state. To accomplish this I was wondering if we should allow developers to add visitors that would be run after all the listeners run. So PlanOptimizer would change to have a new method:
{code}
addStatusResettingVisitor(Visitor v) {
resetters.add(v);
}
{code}
and in the optimize loop
{code}
for (OperatorPlan m : matches) {
if (transformer.check(m)) {
sawMatch = true;
transformer.transform(m);
for(PlanTransformListener l: listeners) {
l.transformed(plan, transformer.reportChanges());
}
}
}
{code}
would change to be:
{code}
for (OperatorPlan m : matches) {
if (transformer.check(m)) {
sawMatch = true;
transformer.transform(m);
for(PlanTransformListener l: listeners) {
l.transformed(plan, transformer.reportChanges());
}
for(Visitor v : resetters) {
v.visit();
}
}
}
{code}
Thoughts?
4) There is not clarity on how column pruning will work in the new optimizer. Will it be represented by a rule? If so, how, since the new optimizer does not allow matching on any operator just on specific operators? Would it be better instead to have it use the Transformers but not the PlanOptimizer infrastructure, since it isn't clear that we would want the column pruning rule to be triggered more than once? To answer these I think we should prototype the column pruning soon. It was one of the hardest parts of the existing infrastructure. We want to make sure it can be done well in this new approach before committing to the approach.
5) The comment was made that while the examples in the document appear to show that the proposal will work for nested plans (that is, inner plans in foreach) they do not show that it will work for operators not yet nestable in foreach (e.g. group, foreach). Since a stated goal of Pig Latin is to someday allow arbitrary nesting, we should validate that the proposal will support these additional operators to be nested in foreach.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ying He updated PIG-1178:
-------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895339#action_12895339 ]
Daniel Dai commented on PIG-1178:
---------------------------------
Still working on a couple of areas:
1. Exception handling
2. Schema merging (LOLoad / LOUnion)
3. One issue in map key pruning, which will prune the map even the map itself is referred in the script
4. One issue in Foreach statement inside nested foreach plan
5. Migrate all old rules, add some new rules (most patches are already there in separate Jiras)
6. Swap the flag to use new logical optimizer by default, testing/bug fixing
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845173#action_12845173 ]
Daniel Dai commented on PIG-1178:
---------------------------------
pig_1178_3.3.patch committed. Manual unit pass.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895771#action_12895771 ]
Daniel Dai commented on PIG-1178:
---------------------------------
PIG-1178-6.patch committed.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: pig_1178_3.4.patch
pig_1178_3.4.patch include all logical relational operators.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Attachment: expressions-2.patch
New patch that addresses the unit test failure and javadoc warnings.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: PIG-1178-4.patch
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: (was: pig_1178_3.2.patch)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845023#action_12845023 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12438710/pig_1178_3.2.patch
against trunk revision 922664.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 28 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/250/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/250/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/250/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800461#action_12800461 ]
Daniel Dai commented on PIG-1178:
---------------------------------
LogicalSchema.size() return null, I assume it is not finished. Otherwise, +1.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798855#action_12798855 ]
Daniel Dai commented on PIG-1178:
---------------------------------
Couple of questions:
1. Why do we need a pos arguments in PlanEdge? What's the use case for that?
2. Where will relational operator methods go? Such as getRequiredFields, getProjectionMap, getRelevantInputs, pruneColumns. Are we going to solve them using uid?
3. What is the functional division between Rule.match() and PatternMatchOperatorPlan.check()? Can we wrap both logic in one class (Rule) rather than two? Leave PatternMatchOperatorPlan simple seems to more clear to the rule writers.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Attachments: lp.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Modi updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800125#action_12800125 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12430202/expressions.patch
against trunk revision 898497.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/174/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/174/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/174/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Modi updated PIG-1178:
----------------------------
Attachment: pig_1178_3.patch
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798943#action_12798943 ]
Daniel Dai commented on PIG-1178:
---------------------------------
Thanks for clarification. +1 for the change.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: lp.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: pig_1178_3.2.patch
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800434#action_12800434 ]
Hadoop QA commented on PIG-1178:
--------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12430285/expressions-2.patch
against trunk revision 898497.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/175/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/175/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/175/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Modi updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ying He updated PIG-1178:
-------------------------
Status: Open (was: Patch Available)
attached a new patch
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799859#action_12799859 ]
Ying He commented on PIG-1178:
------------------------------
a couple questions on expression operators:
1. in ProjectExpression, is it better to change the object variable "input" from "int" to "LogicalRelationalOperator" to point to the operator that the project expression operates on directly? And I don't understand why this operator needs alias it references. But if we change input to operator object, the alias can be get from the operator.
2. I don't know the purpose of ColumnExpression. Is it to capture operands? It doesn't seem to have any special features. So I am not sure if it is necessary
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Attachment: (was: expressions.patch)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798872#action_12798872 ]
Alan Gates commented on PIG-1178:
---------------------------------
bq. 1. Why do we need a pos arguments in PlanEdge? What's the use case for that?
One of the issues we saw frequently with the current implementation of OperatorPlan is that for nodes with multiple inputs (or outputs), if a transformation required disconnecting one of those inputs and connecting a new one, it often changed the order of the inputs (that is, what had been plan.getPredecessors(op).get(1) became plan.getPredecessors(op).get(1)). The ability to connect a PlanEdge as a particular input or output is meant to address this.
bq. 2. Where will relational operator methods go? Such as getRequiredFields, getProjectionMap, getRelevantInputs, pruneColumns. Are we going to solve them using uid?
They should go away. Patching up a plan after the transform will be the responsibility of the PlanTransformListeners. The hypothesis is that schema plus uid will be sufficient for these to do their jobs.
pruneColumns is a special case, but again I think that schema plus uid will be sufficient.
bq. 3. What is the functional division between Rule.match() and PatternMatchOperatorPlan.check()? Can we wrap both logic in one class (Rule) rather than two? Leave PatternMatchOperatorPlan simple seems to more clear to the rule writers.
I'll leave this one for Ying.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Attachments: lp.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Modi updated PIG-1178:
----------------------------
Attachment: pig_1178.patch
Attaching another patch with end-to-end functionality of load,filter,join,store and a few other expression operators.
This patch is self sufficient and can be applied directly on SVN Trunk.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Modi updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
Resubmitting patch again due to core test failures
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: pig_1178_3.4.patch
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895334#action_12895334 ]
Daniel Dai commented on PIG-1178:
---------------------------------
PIG-1178-5.patch committed.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832328#action_12832328 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12435054/pig_1178.patch
against trunk revision 908324.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 34 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 489 release audit warnings (more than the trunk's current 487 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/199/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/199/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/199/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/199/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799371#action_12799371 ]
Alan Gates commented on PIG-1178:
---------------------------------
This code didn't make any changes in the distinct bag area. Also, these same failures have been seen on the load-store branch. And running the test locally does not reproduce the failure. So I suspect there is some intermittent bug with the bags rather than an issue with this code. I'm going to go ahead and check it in.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ying He updated PIG-1178:
-------------------------
Attachment: lp.patch
initial drop for new logical plan framework
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Attachments: lp.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895165#action_12895165 ]
Daniel Dai commented on PIG-1178:
---------------------------------
Did some restructure and bug fixing. Also move package from experimental to newplan.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892960#action_12892960 ]
Alan Gates commented on PIG-1178:
---------------------------------
12K lines of code, wow!
In ProjectExpression, what does the new attachedRelationalOp do? javadoc comments on that in the constructor would be good.
Is the purpose of this patch to also make this the default optimizer or leave it in experimental mode? If it is to make it the default, I think we should move it from the experimental package, though probably in a separate patch. If it isn't, any thoughts on when it would be ready to become the default optimizer?
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837062#action_12837062 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12436606/pig_1178_2.patch
against trunk revision 912064.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 25 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/211/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/211/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/211/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Modi updated PIG-1178:
----------------------------
Attachment: pig_1178.patch
This is a new patch that can be applied to SVN Trunk.
It includes ForEach, InnerLoad, Generate operators along with some LogicalExpression.
It also includes a new optimizer Rule for pushing FilterAboveForeach
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: PIG-1178-10.patch
Patch PIG-1178-10.patch address foreach user defined schema.
test-patch result:
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
All test pass.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-10.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, PIG-1178-8.patch, PIG-1178-9.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
Checked in Ankit's pig_1178.patch.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904485#action_12904485 ]
Daniel Dai commented on PIG-1178:
---------------------------------
PIG-1178-9.patch committed.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, PIG-1178-8.patch, PIG-1178-9.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: PIG-1178-7.patch
PIG-1178-7.patch switch the flag to use new logical plan by default. It fix most unit test except:
1. TestMultiQuery.testMultiQueryJiraPig1169, it depends on PIG-1514, will be fixed automatically once PIG-1514 check in
2. TestPruneColumn.testMapKey3
Both test cases are temporarily commented out. All other unit tests pass.
Here is test-patch result:
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 36 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Modi updated PIG-1178:
----------------------------
Attachment: pig_1178_2.patch
Another patch with a few more LogicalExpressions and some more unit tests using the foreach operator
It also has a rudimentry planPrinter to print new logical plan.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903803#action_12903803 ]
Daniel Dai commented on PIG-1178:
---------------------------------
test-patch result for PIG-11780-8:
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
Patch committed.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, PIG-1178-8.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891438#action_12891438 ]
Daniel Dai commented on PIG-1178:
---------------------------------
Attach PIG-1178-4.patch, include change of the following area:
1. Add all the relational operators
2. Add foreach nested plans
3. Add field schema to expression operators
4. Remove UidStamp, instead, uid will be generated and cached first time we get fieldschema
5. Fix column pruner and all other new logical plan test cases
6. Add TypeCastInserter
Still polishing and refactory the code.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800479#action_12800479 ]
Alan Gates commented on PIG-1178:
---------------------------------
I've checked in the expressions-2.patch. I'll flesh out LogicalSchema in a separate patch.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798112#action_12798112 ]
Alan Gates commented on PIG-1178:
---------------------------------
Comments on lp.patch:
Is the OperatorPlan that is returned by Rule.Match distinct from the plan that the Transformer is operating on? Will the nodes inside that plan be the same nodes (ie that same java objects) that are in the original plan passed to Rule.Match? If the answer is yes and yes, then things will still work, but rule writers will have to be very careful when working with the plans. The reason is that the plan returned by Rule.Match will not be the same plan as returned by Operator.getPlan() when called on the matched nodes. And rules should use the original plan (the one returned by Operator.getPlan()) to navigate, as that is the real plan. I don't see a better way to do this, so I'm ok with it. But we will have to document it very well in the javadocs for writing rules.
In TestExperimentalRule I think we should add tests for two more patterns:
1. a join with two loads as predecessors
2. a spilt with two filters as successors
Also, since I wrote some of this code, we need to have another committers review it before I check it in.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Attachments: lp.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: PIG-1178-6.patch
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
Checked in pig_1178_2.patch
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855486#action_12855486 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12441315/pig_1178_3.4.patch
against trunk revision 932472.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to cause Findbugs to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/281/testReport/
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/281/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832072#action_12832072 ]
Alan Gates commented on PIG-1178:
---------------------------------
Responses to Ashutosh's questions and comments:
bq. b) At various places Runtime Exceptions are thrown. Do we need a new Exception class something like ExperimentalOptimizerException so as to easily spot those exceptions? That will aid in debugging. Also error messages are pretty terse. More details can be put in messages, again to aid in debugging later on.
The error messages are sparse. Part of moving this from prototype to production will be fleshing out those error messages, adding error codes, etc.
As for adding a new exception, I'm not sure I see the value. Hopefully RuntimeException is only used in appropriate places. If it's being used where a different exception should be used, then we can change it.
bq. c) At couple of places, changes are made outside of experimental package. Will be useful to put comments there for why are those needed.
Changes were made to FuncSpec and FileSpec to give them equals functions so that comparisons can be done between two nodes that both contain a FuncSpec. For example, we may want to determine if two load operators are the same. Part of that will be determining if they use the same load function and load the same file.
bq. Was wondering about different optimizations that we do on a complied MR plan. ... [E]ssentially those optimizations are also done through visitors and would benefit greatly if there is a framework for them just as there is one for front-end. Is there any plan to also subsume those visitors (possibly by rewriting them as rule-transform pairs) in this new optimizer or will they be dealt with separately later on?
Plan is too strong a word; there is a hope. I agree that the MR optimizer is a mess, and it needs a framework. The question is whether the same framework will suffice for both. I hope that it will. But to avoid taking on too much we have defined the scope of this current work to just be the logical optimizer. After we have the logical optimizer in good shape we will need to address issues in the MR optimizer. Hopefully this framework being developed will work with minimal refactoring and changes. If not, a different framework will need to be designed for the MR case.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832313#action_12832313 ]
Ying He commented on PIG-1178:
------------------------------
Here is my thoughts to use this framework to implement PruneColumns.
1. Separate prune columns and prune map keys into 2 rules. Current implementation mixed them in one class. It's better to separate them to make each rule simpler.
2. The prune column rule can be implemented by creating a new visitor. This visitor is called from transform(), and it visits every LogicalRelationalOperator by reverse dependency order. Each visit(LogicalRelationalOperator) calculates the required output uids by combining the input uids from it successors. If a node is the sink of the plan, the output uids are retrieved from its schema. The input uids are calculated from its output uids by looking into the expression plan(s) of this operator. If an output uid is derived from other uids, the source uids should be put into input uids. For example, a+b is from a & b. The input uids should keep the uid of a & b. Each operator should consider its logical meanings when calculating input uids from output uids. For example, for LOCross, the input uids should contain at least one field from each input.
The input uids and output uids can be added into the operator as annotations.
3. After step 2, use another visitor to go over the plan again by dependency order to prune the columns. This can be done by reading out the input and output uids for each node.
4. I think it's ok to implement prune column and prune map key as regular rule. They just need to overwrite the match().
public List<OperatorPlan> match(OperatorPlan plan) {
List<OperatorPlan> ll = new ArrayList<OperatorPlan();
ll.add(plan);
return ll;
}
This method tells optimizer that only one match is find, which is the plan itself.
5. For Transformer class, I suggest to get rid of check() and change void transform() into boolean transform(). If transform() returns false, it means no transformation is made. If it returns true, transformation is made. The reason is that for some rules, it is not easy to know if a change is going to be made, such as PruneColumn rule. If we have both check() and transform(), lots of logic would be duplicated in these two methods.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907061#action_12907061 ]
Daniel Dai commented on PIG-1178:
---------------------------------
PIG-1178-11.patch committed to both trunk and 0.8 branch.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-10.patch, PIG-1178-11.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, PIG-1178-8.patch, PIG-1178-9.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Modi updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: PIG-1178-9.patch
Update help message.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, PIG-1178-8.patch, PIG-1178-9.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901543#action_12901543 ]
Daniel Dai commented on PIG-1178:
---------------------------------
PIG-1178-7.patch committed.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1178:
--------------------------------
Fix Version/s: 0.8.0
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837954#action_12837954 ]
Ankit Modi commented on PIG-1178:
---------------------------------
the core tests are failing due to some issue with hudson or the framework.
I ran the core tests again yesterday night and they passed.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832164#action_12832164 ]
Ying He commented on PIG-1178:
------------------------------
for the annotation resetting, I think it can be implemented as a PlanTransformListener. The listener has access to the plan and can reset every node, given the order is not important.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844843#action_12844843 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12438661/pig_1178_3.patch
against trunk revision 922169.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 28 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 89 javac compiler warnings (more than the trunk's current 87 warnings).
-1 findbugs. The patch appears to introduce 3 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/248/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/248/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/248/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803597#action_12803597 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12430918/lp.patch
against trunk revision 901900.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/170/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/170/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/170/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828895#action_12828895 ]
Ashutosh Chauhan commented on PIG-1178:
---------------------------------------
Last patch was more then 5500 lines. Phew! To make it easier to read, I have few minor comments which you probably can sneak in your next upload:
a) There is commented code at various places. Deleting those lines will make patch smaller.
b) At various places Runtime Exceptions are thrown. Do we need a new Exception class something like ExperimentalOptimizerException so as to easily spot those exceptions? That will aid in debugging. Also error messages are pretty terse. More details can be put in messages, again to aid in debugging later on.
c) At couple of places, changes are made outside of experimental package. Will be useful to put comments there for why are those needed.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831886#action_12831886 ]
Ashutosh Chauhan commented on PIG-1178:
---------------------------------------
Was wondering about different optimizations that we do on a complied MR plan. Not sure if its already been discussed or is in some doc. But essentially those optimizations are also done through visitors and would benefit greatly if there is a framework for them just as there is one for front-end. Is there any plan to also subsume those visitors (possibly by rewriting them as rule-transform pairs) in this new optimizer or will they be dealt with separately later on?
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: pig_1178_3.2.patch
Address findbugs warnings.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ying He updated PIG-1178:
-------------------------
Attachment: lp.patch
patch to add relational operator, optimization rules and logical plan migration visitor
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837196#action_12837196 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12436606/pig_1178_2.patch
against trunk revision 915079.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 25 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/216/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/216/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/216/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12906592#action_12906592 ]
Daniel Dai commented on PIG-1178:
---------------------------------
Patch PIG-1178-10.patch committed.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-10.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, PIG-1178-8.patch, PIG-1178-9.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799071#action_12799071 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429960/PIG_1178.patch
against trunk revision 896951.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/170/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/170/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/170/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: PIG-1178-8.patch
PIG-1178-8.patch fix TestPruneColumn.testMapKey3
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, PIG-1178-7.patch, PIG-1178-8.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799552#action_12799552 ]
Hadoop QA commented on PIG-1178:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12430056/expressions.patch
against trunk revision 898497.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/171/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/171/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/171/console
This message is automatically generated.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798880#action_12798880 ]
Ying He commented on PIG-1178:
------------------------------
the Rule.match() finds a potential match and delegate to PatternMatchOperatorPlan to further verify if it matches the pattern. During it's verification, the plan is filled with the operators from original plan. The PetternMatchOperatorPlan is not visible to rule writers. Rule writers should only use OperatorPlan to operate on the matched sub-plans.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: lp.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, PIG-1178-6.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Ying He (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798273#action_12798273 ]
Ying He commented on PIG-1178:
------------------------------
yes, the operator plan that Rule.match returned has the same java objects as the original plan. So as Alan said, it's a bit confusing. the plan returned by Rule.match is more similar to a view.
the test cases for the two patterns will be added later.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Attachments: lp.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
Resetting patch available so Hudson will rerun the tests.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799372#action_12799372 ]
Alan Gates commented on PIG-1178:
---------------------------------
Checked in PIG_1178.patch
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: pig_1178_3.3.patch
Address additional findbugs warnings
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
Resetting patch status to get hudson to rerun tests with new expression.patch
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Attachment: expressions.patch
Second take on expressions.patch, with fixes for the findbugs warning and Ying's comment on not having aliases in projections.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai reassigned PIG-1178:
-------------------------------
Assignee: Daniel Dai (was: Ying He)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Attachment: (was: pig_1178_3.4.patch)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803565#action_12803565 ]
Alan Gates commented on PIG-1178:
---------------------------------
Comments on lp.patch:
1) In LOJoin.getSchema, these lines of code:
{code}
for (Operator op : inputs) {
LogicalSchema inputSchema = ((LogicalRelationalOperator)op).getSchema();
// the schema of one input is unknown, so the join schema is unknown, just return
if (inputSchema == null) {
schemaSet = true;
return schema;
}
{code}
You are assuming that schema is null. It would be better to explicitly set schema to null and then return it.
2) In SplitFilter.transform you put it in a while loop, finding each 'and' and splitting it into another filter. But there's already an outer while loop (the one in the optimizer applying the rule over and over) that will do that. One of the assertions in this design is that each rule should be as simple as possible. This rule should just split one and, and let the next application of the rule find the next and and split it again.
Same comment applies to MergeFilter.transform and to PushUpFilterTransformer.check and .transform.
3) In MergeFilter.check: IIRC implicit splits aren't inserted into the plan until the logical to physical transformation. So it's possible that a filter actually has multiple successors. So instead of:
{code}
if (succeds != null && succeds.size()>0) {
if (succeds.get(0) instanceof LOFilter) {
return true;
}
}
{code}
it should read
{code}
if (succeds != null && succeds.size() == 1) {
if (succeds.get(0) instanceof LOFilter) {
return true;
}
}
{code}
4) In MergeFilter.combineFilterCond: The expressions have been written in such a way that they manage their own connections when they are created. See for example, AndExpression. In its constructor it takes it add itself to the expression plan and connects itself to its two operands. So there no need to to do the addPlan.add and addPlan.connect calls.
5) In PushUpFilterTransformer.check, you need to check that the join type is inner. Pushing past outer joins is much trickier, and need not be handled here.
6) In PushUpFilterTransformer.check I don't understand what findCommon is doing. In any case, it should not be paying attention to aliases. It should be using the inputNums from the projection. It should be checking that all projections in the filter are associated with the same inputNum. If so, it is pushable to that inputNum. If not, not. In the same way transform should be using inputNum to find the right predecessor, not aliases.
7) We need a fourth rule to handle swapping filters, so each one can be tried against the join. Since this rule will always pass check (it would just be two filters in a row) we need a way to check that it doesn't run more than twice for a given pair of filters. We can accomplish this by having it 'sign' each filter in the node each time it is applied. This is what the annotate call on Operator is for. So each time the transform is applied, it would annotate both filters with info that it was applied, and to which filters. Then part of check can be two check that this rule has been applied at most twice.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Patch Available (was: Open)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1178:
----------------------------
Status: Open (was: Patch Available)
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Daniel Dai
> Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates reassigned PIG-1178:
-------------------------------
Assignee: Ying He
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: lp.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1178) LogicalPlan and Optimizer are too
complex and hard to work with
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1178:
----------------------------
Attachment: expressions.patch
A patch to flesh out a few expressions in the prototype. Also makes minor changes to Schema, LogicalRelationalOperator, and the walkers to address a few oversights.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions.patch, lp.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.