You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Aniket Mokashi (Created) (JIRA)" <ji...@apache.org> on 2012/01/10 08:18:40 UTC
[jira] [Created] (HIVE-2702) listPartitionsByFilter only supports
non-string partitions
listPartitionsByFilter only supports non-string partitions
----------------------------------------------------------
Key: HIVE-2702
URL: https://issues.apache.org/jira/browse/HIVE-2702
Project: Hive
Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Aniket Mokashi
listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
//Can only support partitions whose types are string
if( ! table.getPartitionKeys().get(partitionColumnIndex).
getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
throw new MetaException
("Filtering is supported only on partition keys of type string");
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2702) listPartitionsByFilter only supports
non-string partitions
Posted by "Carl Steinbach (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach reassigned HIVE-2702:
------------------------------------
Assignee: Aniket Mokashi
> listPartitionsByFilter only supports non-string partitions
> ----------------------------------------------------------
>
> Key: HIVE-2702
> URL: https://issues.apache.org/jira/browse/HIVE-2702
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Aniket Mokashi
> Assignee: Aniket Mokashi
> Attachments: HIVE-2702.1.patch, HIVE-2702.D2043.1.patch
>
>
> listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
> //Can only support partitions whose types are string
> if( ! table.getPartitionKeys().get(partitionColumnIndex).
> getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
> throw new MetaException
> ("Filtering is supported only on partition keys of type string");
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2702) listPartitionsByFilter only supports
non-string partitions
Posted by "Aniket Mokashi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aniket Mokashi updated HIVE-2702:
---------------------------------
Status: Patch Available (was: Open)
> listPartitionsByFilter only supports non-string partitions
> ----------------------------------------------------------
>
> Key: HIVE-2702
> URL: https://issues.apache.org/jira/browse/HIVE-2702
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Aniket Mokashi
> Attachments: HIVE-2702.1.patch
>
>
> listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
> //Can only support partitions whose types are string
> if( ! table.getPartitionKeys().get(partitionColumnIndex).
> getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
> throw new MetaException
> ("Filtering is supported only on partition keys of type string");
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2702) listPartitionsByFilter only supports
non-string partitions
Posted by "Aniket Mokashi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aniket Mokashi updated HIVE-2702:
---------------------------------
Attachment: HIVE-2702.1.patch
> listPartitionsByFilter only supports non-string partitions
> ----------------------------------------------------------
>
> Key: HIVE-2702
> URL: https://issues.apache.org/jira/browse/HIVE-2702
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Aniket Mokashi
> Attachments: HIVE-2702.1.patch
>
>
> listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
> //Can only support partitions whose types are string
> if( ! table.getPartitionKeys().get(partitionColumnIndex).
> getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
> throw new MetaException
> ("Filtering is supported only on partition keys of type string");
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2702) listPartitionsByFilter only supports
string partitions
Posted by "Thomas Weise (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Weise updated HIVE-2702:
-------------------------------
Summary: listPartitionsByFilter only supports string partitions (was: listPartitionsByFilter only supports non-string partitions)
> listPartitionsByFilter only supports string partitions
> ------------------------------------------------------
>
> Key: HIVE-2702
> URL: https://issues.apache.org/jira/browse/HIVE-2702
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Aniket Mokashi
> Assignee: Aniket Mokashi
> Attachments: HIVE-2702.1.patch, HIVE-2702.D2043.1.patch
>
>
> listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
> //Can only support partitions whose types are string
> if( ! table.getPartitionKeys().get(partitionColumnIndex).
> getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
> throw new MetaException
> ("Filtering is supported only on partition keys of type string");
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2702) listPartitionsByFilter only supports
non-string partitions
Posted by "Ashutosh Chauhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HIVE-2702:
-----------------------------------
Status: Open (was: Patch Available)
> listPartitionsByFilter only supports non-string partitions
> ----------------------------------------------------------
>
> Key: HIVE-2702
> URL: https://issues.apache.org/jira/browse/HIVE-2702
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Aniket Mokashi
> Assignee: Aniket Mokashi
> Attachments: HIVE-2702.1.patch, HIVE-2702.D2043.1.patch
>
>
> listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
> //Can only support partitions whose types are string
> if( ! table.getPartitionKeys().get(partitionColumnIndex).
> getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
> throw new MetaException
> ("Filtering is supported only on partition keys of type string");
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2702) listPartitionsByFilter only supports
non-string partitions
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2702:
------------------------------
Attachment: HIVE-2702.D2043.1.patch
aniket486 requested code review of "HIVE-2702 [jira] listPartitionsByFilter only supports non-string partitions".
Reviewers: JIRA
https://issues.apache.org/jira/browse/HIVE-2702
Added support for integer partitions in listPartitionByFilter method
listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
//Can only support partitions whose types are string
if( ! table.getPartitionKeys().get(partitionColumnIndex).
getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
throw new MetaException
("Filtering is supported only on partition keys of type string");
}
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D2043
AFFECTED FILES
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/4395/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> listPartitionsByFilter only supports non-string partitions
> ----------------------------------------------------------
>
> Key: HIVE-2702
> URL: https://issues.apache.org/jira/browse/HIVE-2702
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Aniket Mokashi
> Attachments: HIVE-2702.1.patch, HIVE-2702.D2043.1.patch
>
>
> listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
> //Can only support partitions whose types are string
> if( ! table.getPartitionKeys().get(partitionColumnIndex).
> getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
> throw new MetaException
> ("Filtering is supported only on partition keys of type string");
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2702) listPartitionsByFilter only supports
string partitions
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438357#comment-13438357 ]
Jakob Homan commented on HIVE-2702:
-----------------------------------
This is very limiting for HCat as it means HCat clients cannot access non-string partitions from Pig or MR, whereas they are available via Hive directly. The JDOQL that's used in generateJDOFilter() to generate the query is not powerful enough to support stripping out the value, casting it to a number, and then sorting it thusly. The best solution is probably to re-write ObjectStore.listPartitionNamesByFilter() to not use the Partitions table (which stores the values as "PARTITION=FOO" but rather take advantage of PARTITION_KEY_VALS, which stores the partition values by themselves. Any objections to this approach?
> listPartitionsByFilter only supports string partitions
> ------------------------------------------------------
>
> Key: HIVE-2702
> URL: https://issues.apache.org/jira/browse/HIVE-2702
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Aniket Mokashi
> Assignee: Aniket Mokashi
> Attachments: HIVE-2702.1.patch, HIVE-2702.D2043.1.patch
>
>
> listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
> //Can only support partitions whose types are string
> if( ! table.getPartitionKeys().get(partitionColumnIndex).
> getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
> throw new MetaException
> ("Filtering is supported only on partition keys of type string");
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2702) listPartitionsByFilter only supports
non-string partitions
Posted by "Aniket Mokashi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232319#comment-13232319 ]
Aniket Mokashi commented on HIVE-2702:
--------------------------------------
I agree on your comment, I didn't catch it earlier. I wonder if there is an elegant workaround for this.
> listPartitionsByFilter only supports non-string partitions
> ----------------------------------------------------------
>
> Key: HIVE-2702
> URL: https://issues.apache.org/jira/browse/HIVE-2702
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Aniket Mokashi
> Assignee: Aniket Mokashi
> Attachments: HIVE-2702.1.patch, HIVE-2702.D2043.1.patch
>
>
> listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
> //Can only support partitions whose types are string
> if( ! table.getPartitionKeys().get(partitionColumnIndex).
> getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
> throw new MetaException
> ("Filtering is supported only on partition keys of type string");
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2702) listPartitionsByFilter only supports
non-string partitions
Posted by "Ashutosh Chauhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230971#comment-13230971 ]
Ashutosh Chauhan commented on HIVE-2702:
----------------------------------------
@Aniket,
This is by design. Partition values are stored as strings in backend db (mysql) so pushing filters into db where partition column is of numeric types won't work, since then comparison will happen lexicographically. You should be able to catch this with rigorous tests. e.g., with your patch on, create table with partition key of int type, add partitions 1-11 and then do filter p < 2 and you will get partitions 1,10,11 instead of 1. Though, you can still push equality predicate.
Enabling this feature requires mysql table schema updates which can retain type information for partition keys.
> listPartitionsByFilter only supports non-string partitions
> ----------------------------------------------------------
>
> Key: HIVE-2702
> URL: https://issues.apache.org/jira/browse/HIVE-2702
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Aniket Mokashi
> Assignee: Aniket Mokashi
> Attachments: HIVE-2702.1.patch, HIVE-2702.D2043.1.patch
>
>
> listPartitionsByFilter supports only non-string partitions. This is because its explicitly specified in generateJDOFilterOverPartitions in ExpressionTree.java.
> //Can only support partitions whose types are string
> if( ! table.getPartitionKeys().get(partitionColumnIndex).
> getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
> throw new MetaException
> ("Filtering is supported only on partition keys of type string");
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira