You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Ashutosh Chauhan (JIRA)" <ji...@apache.org> on 2011/02/25 23:45:21 UTC
[jira] Created: (PIG-1871) Dont throw exception if partition
filters cannot be pushed up.
Dont throw exception if partition filters cannot be pushed up.
----------------------------------------------------------------
Key: PIG-1871
URL: https://issues.apache.org/jira/browse/PIG-1871
Project: Pig
Issue Type: Bug
Reporter: Ashutosh Chauhan
Priority: Minor
Instead don't try to push partition filters up and continue execution.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-1871) Dont throw exception if partition
filters cannot be pushed up.
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich reassigned PIG-1871:
-----------------------------------
Assignee: Richard Ding
> Dont throw exception if partition filters cannot be pushed up.
> ----------------------------------------------------------------
>
> Key: PIG-1871
> URL: https://issues.apache.org/jira/browse/PIG-1871
> Project: Pig
> Issue Type: Bug
> Reporter: Ashutosh Chauhan
> Assignee: Richard Ding
> Priority: Minor
> Fix For: 0.9.0
>
>
> Instead don't try to push partition filters up and continue execution.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (PIG-1871) Dont throw exception if partition
filters cannot be pushed up.
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999643#comment-12999643 ]
Dmitriy V. Ryaboy commented on PIG-1871:
----------------------------------------
Agreed with the spirit of the thing, but for the particular example -- why not push up the ds and action filters as (ds < '20091103' or action == 'continue') and then apply the original filters in Pig? You still get the benefits of whatever partition pushdown can do, and don't sacrifice correctness.
> Dont throw exception if partition filters cannot be pushed up.
> ----------------------------------------------------------------
>
> Key: PIG-1871
> URL: https://issues.apache.org/jira/browse/PIG-1871
> Project: Pig
> Issue Type: Bug
> Reporter: Ashutosh Chauhan
> Priority: Minor
> Fix For: 0.9.0
>
>
> Instead don't try to push partition filters up and continue execution.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-1871) Dont throw exception if partition
filters cannot be pushed up.
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021132#comment-13021132 ]
Daniel Dai commented on PIG-1871:
---------------------------------
+1
> Dont throw exception if partition filters cannot be pushed up.
> ----------------------------------------------------------------
>
> Key: PIG-1871
> URL: https://issues.apache.org/jira/browse/PIG-1871
> Project: Pig
> Issue Type: Bug
> Reporter: Ashutosh Chauhan
> Assignee: Richard Ding
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: PIG-1871_1.patch
>
>
> Instead don't try to push partition filters up and continue execution.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (PIG-1871) Dont throw exception if partition
filters cannot be pushed up.
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1871:
--------------------------------
Fix Version/s: 0.9.0
> Dont throw exception if partition filters cannot be pushed up.
> ----------------------------------------------------------------
>
> Key: PIG-1871
> URL: https://issues.apache.org/jira/browse/PIG-1871
> Project: Pig
> Issue Type: Bug
> Reporter: Ashutosh Chauhan
> Priority: Minor
> Fix For: 0.9.0
>
>
> Instead don't try to push partition filters up and continue execution.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (PIG-1871) Dont throw exception if partition
filters cannot be pushed up.
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999654#comment-12999654 ]
Ashutosh Chauhan commented on PIG-1871:
---------------------------------------
In Pig 0.8 following succeeds:
a = load 'mydata' USING MyLoader();
b0 = filter a BY
( (ds < '20091103' )
or (action == 'continue' )
);
c = foreach b0 generate ds, action, age, data;
store c into ':OUTPATH:';
This implies that in 0.8, b0 and b got combined first. Then, Pig tried to weed out partitioning columns from the combined expression and bailed out. But clearly, b0 can be pushed up and is separated by 'AND' in combined expression so should have been pushed up. There will still be the cases where partition columns can't be weeded out easily from an expression. In those cases, don't try to push up anything and continue.
> Dont throw exception if partition filters cannot be pushed up.
> ----------------------------------------------------------------
>
> Key: PIG-1871
> URL: https://issues.apache.org/jira/browse/PIG-1871
> Project: Pig
> Issue Type: Bug
> Reporter: Ashutosh Chauhan
> Priority: Minor
> Fix For: 0.9.0
>
>
> Instead don't try to push partition filters up and continue execution.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-1871) Dont throw exception if partition
filters cannot be pushed up.
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding resolved PIG-1871.
-------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Patch committed to trunk.
> Dont throw exception if partition filters cannot be pushed up.
> ----------------------------------------------------------------
>
> Key: PIG-1871
> URL: https://issues.apache.org/jira/browse/PIG-1871
> Project: Pig
> Issue Type: Bug
> Reporter: Ashutosh Chauhan
> Assignee: Richard Ding
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: PIG-1871_1.patch
>
>
> Instead don't try to push partition filters up and continue execution.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (PIG-1871) Dont throw exception if partition
filters cannot be pushed up.
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999630#comment-12999630 ]
Ashutosh Chauhan commented on PIG-1871:
---------------------------------------
a = load 'mydata' USING MyLoader();
b0 = filter a BY
( (ds < '20091103' )
or (action == 'continue' )
);
b = filter b0 BY
( (ds < '20091103' and age < 50)
or (action == 'continue' and age > 100)
);
c = foreach b generate ds, action, age, data;
store c into ':OUTPATH:';
...
fails with message:
Grunt - ERROR 1112:
Unsupported query:
You have an partition column (datestamp ) in a construction like: (pcond and ...) or (pcond and ...) where pcond is a condition on a partition column.
In this testcase ds and action are partition columns while age is not. So, pig decided it cannot split the filter and can't push the partition filter to the loader. In such a condition instead of dying, Pig should disable partition filter push up and continue.
> Dont throw exception if partition filters cannot be pushed up.
> ----------------------------------------------------------------
>
> Key: PIG-1871
> URL: https://issues.apache.org/jira/browse/PIG-1871
> Project: Pig
> Issue Type: Bug
> Reporter: Ashutosh Chauhan
> Priority: Minor
>
> Instead don't try to push partition filters up and continue execution.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-1871) Dont throw exception if partition
filters cannot be pushed up.
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1871:
------------------------------
Attachment: PIG-1871_1.patch
Attaching patch that wouldn't throw exceptions if partition filter can't be pushed down. Now warnings are logged instead.
> Dont throw exception if partition filters cannot be pushed up.
> ----------------------------------------------------------------
>
> Key: PIG-1871
> URL: https://issues.apache.org/jira/browse/PIG-1871
> Project: Pig
> Issue Type: Bug
> Reporter: Ashutosh Chauhan
> Assignee: Richard Ding
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: PIG-1871_1.patch
>
>
> Instead don't try to push partition filters up and continue execution.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (PIG-1871) Dont throw exception if partition
filters cannot be pushed up.
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999649#comment-12999649 ]
Ashutosh Chauhan commented on PIG-1871:
---------------------------------------
Yes, that is the most appropriate thing to do in this case.
Additionally, The query works as is when using older version of Pig:
Apache Pig version 0.7.0.20.10.0.1006041903 (r951530)
compiled Jun 04 2010, 19:03:37
> Dont throw exception if partition filters cannot be pushed up.
> ----------------------------------------------------------------
>
> Key: PIG-1871
> URL: https://issues.apache.org/jira/browse/PIG-1871
> Project: Pig
> Issue Type: Bug
> Reporter: Ashutosh Chauhan
> Priority: Minor
> Fix For: 0.9.0
>
>
> Instead don't try to push partition filters up and continue execution.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira