You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Xuefu Zhang (JIRA)" <ji...@apache.org> on 2010/08/25 23:36:19 UTC

[jira] Created: (PIG-1567) Optimization rule FilterAboveForeach is too restrictive and doesn't handle project * correctly

Optimization rule FilterAboveForeach is too restrictive and doesn't handle project * correctly
----------------------------------------------------------------------------------------------

                 Key: PIG-1567
                 URL: https://issues.apache.org/jira/browse/PIG-1567
             Project: Pig
          Issue Type: Bug
            Reporter: Xuefu Zhang
             Fix For: 0.8.0


FilterAboveForeach rule is to optimize the plan by pushing up filter above previous foreach operator. However, during code review, two major problems were found:

1. Current implementation assumes that if no projection is found in the filter condition then all columns from foreach are projected. This issue prevents the following optimization:
    	A = LOAD 'file.txt' AS (a(u,v), b, c);
    	B = FOREACH A GENERATE $0, b;
    	C = FILTER B BY 8 > 5;
    	STORE C INTO 'empty';

2. Current implementation doesn't handle * probjection, which means project all columns. As a result, it wasn't able to optimize the following:
    	A = LOAD 'file.txt' AS (a(u,v), b, c);
    	B = FOREACH A GENERATE $0, b;
    	C = FILTER B BY Identity.class.getName(*) > 5;
    	STORE C INTO 'empty';
    	


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (PIG-1567) Optimization rule FilterAboveForeach is too restrictive and doesn't handle project * correctly

Posted by "Xuefu Zhang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xuefu Zhang resolved PIG-1567.
------------------------------

    Resolution: Duplicate

Duplicate of PIG-1568.

> Optimization rule FilterAboveForeach is too restrictive and doesn't handle project * correctly
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1567
>                 URL: https://issues.apache.org/jira/browse/PIG-1567
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Xuefu Zhang
>            Assignee: Xuefu Zhang
>             Fix For: 0.8.0
>
>
> FilterAboveForeach rule is to optimize the plan by pushing up filter above previous foreach operator. However, during code review, two major problems were found:
> 1. Current implementation assumes that if no projection is found in the filter condition then all columns from foreach are projected. This issue prevents the following optimization:
>     	A = LOAD 'file.txt' AS (a(u,v), b, c);
>     	B = FOREACH A GENERATE $0, b;
>     	C = FILTER B BY 8 > 5;
>     	STORE C INTO 'empty';
> 2. Current implementation doesn't handle * probjection, which means project all columns. As a result, it wasn't able to optimize the following:
>     	A = LOAD 'file.txt' AS (a(u,v), b, c);
>     	B = FOREACH A GENERATE $0, b;
>     	C = FILTER B BY Identity.class.getName(*) > 5;
>     	STORE C INTO 'empty';
>     	

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (PIG-1567) Optimization rule FilterAboveForeach is too restrictive and doesn't handle project * correctly

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Olga Natkovich reassigned PIG-1567:
-----------------------------------

    Assignee: Xuefu Zhang

> Optimization rule FilterAboveForeach is too restrictive and doesn't handle project * correctly
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1567
>                 URL: https://issues.apache.org/jira/browse/PIG-1567
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Xuefu Zhang
>            Assignee: Xuefu Zhang
>             Fix For: 0.8.0
>
>
> FilterAboveForeach rule is to optimize the plan by pushing up filter above previous foreach operator. However, during code review, two major problems were found:
> 1. Current implementation assumes that if no projection is found in the filter condition then all columns from foreach are projected. This issue prevents the following optimization:
>     	A = LOAD 'file.txt' AS (a(u,v), b, c);
>     	B = FOREACH A GENERATE $0, b;
>     	C = FILTER B BY 8 > 5;
>     	STORE C INTO 'empty';
> 2. Current implementation doesn't handle * probjection, which means project all columns. As a result, it wasn't able to optimize the following:
>     	A = LOAD 'file.txt' AS (a(u,v), b, c);
>     	B = FOREACH A GENERATE $0, b;
>     	C = FILTER B BY Identity.class.getName(*) > 5;
>     	STORE C INTO 'empty';
>     	

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.