You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2009/01/07 21:04:44 UTC
[jira] Created: (HIVE-218) predicate on partitioning column is
ignored in many places
predicate on partitioning column is ignored in many places
----------------------------------------------------------
Key: HIVE-218
URL: https://issues.apache.org/jira/browse/HIVE-218
Project: Hadoop Hive
Issue Type: Bug
Components: Query Processor
Reporter: Joydeep Sen Sarma
Priority: Blocker
We tried two queries yesterday that bought up several problems:
1. predicate on partitioning column within a join clause was ignored:
FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx AND e.ds=2009-01-05) INSERT OVERWRITE TABLE ...
the plan tried to scan *all* partitions!
2. predicate on partitioning clause inside insert clause was ignored (we took the previous query and moved the partition filter to the insert statement)
FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx ) INSERT OVERWRITE TABLE ... WHERE e.ds=2009-01-05;
the plan again tried to scan *all* partitions
the really bad thing is that we were able to detect this problem only because of metastore inconsistencies - otherwise - we would have merrily scanned all the data. This is really critical to get fixed - because this means that we may actually be scanning tons of unnecessary data in production.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-218) predicate on partitioning column is
ignored in many places
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Thusoo updated HIVE-218:
-------------------------------
Priority: Critical (was: Blocker)
Affects Version/s: 0.3.0
Marking this as a critical as there is a workaround for this by putting the predicate on the partitioning column within a subquery.
In the example above, this can be achieved by
FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 .... JOIN (SELECT aaa.* FROM aaa WHERE aaa.ds-2009-01-05) e ....
> predicate on partitioning column is ignored in many places
> ----------------------------------------------------------
>
> Key: HIVE-218
> URL: https://issues.apache.org/jira/browse/HIVE-218
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Joydeep Sen Sarma
> Assignee: Prasad Chakka
> Priority: Critical
>
> We tried two queries yesterday that bought up several problems:
> 1. predicate on partitioning column within a join clause was ignored:
> FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx AND e.ds=2009-01-05) INSERT OVERWRITE TABLE ...
> the plan tried to scan *all* partitions!
> 2. predicate on partitioning clause inside insert clause was ignored (we took the previous query and moved the partition filter to the insert statement)
> FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx ) INSERT OVERWRITE TABLE ... WHERE e.ds=2009-01-05;
> the plan again tried to scan *all* partitions
> the really bad thing is that we were able to detect this problem only because of metastore inconsistencies - otherwise - we would have merrily scanned all the data. This is really critical to get fixed - because this means that we may actually be scanning tons of unnecessary data in production.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-218) predicate on partitioning column is
ignored in many places
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma reassigned HIVE-218:
--------------------------------------
Assignee: Prasad Chakka
Ashish said this would be resolved by predicate pushdown
> predicate on partitioning column is ignored in many places
> ----------------------------------------------------------
>
> Key: HIVE-218
> URL: https://issues.apache.org/jira/browse/HIVE-218
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Joydeep Sen Sarma
> Assignee: Prasad Chakka
> Priority: Blocker
>
> We tried two queries yesterday that bought up several problems:
> 1. predicate on partitioning column within a join clause was ignored:
> FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx AND e.ds=2009-01-05) INSERT OVERWRITE TABLE ...
> the plan tried to scan *all* partitions!
> 2. predicate on partitioning clause inside insert clause was ignored (we took the previous query and moved the partition filter to the insert statement)
> FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx ) INSERT OVERWRITE TABLE ... WHERE e.ds=2009-01-05;
> the plan again tried to scan *all* partitions
> the really bad thing is that we were able to detect this problem only because of metastore inconsistencies - otherwise - we would have merrily scanned all the data. This is really critical to get fixed - because this means that we may actually be scanning tons of unnecessary data in production.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-218) predicate on partitioning column is
ignored in many places
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Thusoo resolved HIVE-218.
--------------------------------
Resolution: Duplicate
Duplicate of HIVE-578
> predicate on partitioning column is ignored in many places
> ----------------------------------------------------------
>
> Key: HIVE-218
> URL: https://issues.apache.org/jira/browse/HIVE-218
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Joydeep Sen Sarma
> Assignee: Ashish Thusoo
> Priority: Critical
>
> We tried two queries yesterday that bought up several problems:
> 1. predicate on partitioning column within a join clause was ignored:
> FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx AND e.ds=2009-01-05) INSERT OVERWRITE TABLE ...
> the plan tried to scan *all* partitions!
> 2. predicate on partitioning clause inside insert clause was ignored (we took the previous query and moved the partition filter to the insert statement)
> FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx ) INSERT OVERWRITE TABLE ... WHERE e.ds=2009-01-05;
> the plan again tried to scan *all* partitions
> the really bad thing is that we were able to detect this problem only because of metastore inconsistencies - otherwise - we would have merrily scanned all the data. This is really critical to get fixed - because this means that we may actually be scanning tons of unnecessary data in production.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-218) predicate on partitioning column is
ignored in many places
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Chakka reassigned HIVE-218:
----------------------------------
Assignee: Ashish Thusoo (was: Prasad Chakka)
I think HIVE-578 will fix this as well.
> predicate on partitioning column is ignored in many places
> ----------------------------------------------------------
>
> Key: HIVE-218
> URL: https://issues.apache.org/jira/browse/HIVE-218
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Joydeep Sen Sarma
> Assignee: Ashish Thusoo
> Priority: Critical
>
> We tried two queries yesterday that bought up several problems:
> 1. predicate on partitioning column within a join clause was ignored:
> FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx AND e.ds=2009-01-05) INSERT OVERWRITE TABLE ...
> the plan tried to scan *all* partitions!
> 2. predicate on partitioning clause inside insert clause was ignored (we took the previous query and moved the partition filter to the insert statement)
> FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx ) INSERT OVERWRITE TABLE ... WHERE e.ds=2009-01-05;
> the plan again tried to scan *all* partitions
> the really bad thing is that we were able to detect this problem only because of metastore inconsistencies - otherwise - we would have merrily scanned all the data. This is really critical to get fixed - because this means that we may actually be scanning tons of unnecessary data in production.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.