You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by outofmem0ry <gi...@git.apache.org> on 2017/09/15 23:07:27 UTC
[GitHub] incubator-hawq pull request #1287: HAWQ-1527 Enabled partition filtering for...
GitHub user outofmem0ry opened a pull request:
https://github.com/apache/incubator-hawq/pull/1287
HAWQ-1527 Enabled partition filtering for integral data types
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/outofmem0ry/incubator-hawq feature/HAWQ-1527
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/1287.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1287
----
commit 883571eba141d63ff770fa10764842c81f4da01a
Author: Shubham Sharma <sh...@gmail.com>
Date: 2017-09-15T23:05:19Z
HAWQ-1527 Added feature to enable partition filtering for integral data types
----
---
[GitHub] incubator-hawq issue #1287: HAWQ-1527 Enabled partition filtering for integr...
Posted by outofmem0ry <gi...@git.apache.org>.
Github user outofmem0ry commented on the issue:
https://github.com/apache/incubator-hawq/pull/1287
Thanks @shivzone.
@sansanichfb Could you review this as well.
---
[GitHub] incubator-hawq issue #1287: HAWQ-1527 Enabled partition filtering for integr...
Posted by radarwave <gi...@git.apache.org>.
Github user radarwave commented on the issue:
https://github.com/apache/incubator-hawq/pull/1287
There are already two approved reviews. I will merge this PR soon.
---
[GitHub] incubator-hawq pull request #1287: HAWQ-1527 Enabled partition filtering for...
Posted by outofmem0ry <gi...@git.apache.org>.
Github user outofmem0ry closed the pull request at:
https://github.com/apache/incubator-hawq/pull/1287
---
[GitHub] incubator-hawq pull request #1287: HAWQ-1527 Enabled partition filtering for...
Posted by lavjain <gi...@git.apache.org>.
Github user lavjain commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1287#discussion_r141147551
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenter.java ---
@@ -418,8 +427,15 @@ private boolean buildSingleFilter(Object filter,
return false;
}
- if (!partitionkeyTypes.get(filterColumnName).equalsIgnoreCase(serdeConstants.STRING_TYPE_NAME)) {
- LOG.debug("Filter type is not string type , ignore this filter for hive: "
+ /*
+ * HAWQ-1527 - Filtering only supported for partition columns of type string or
+ * intgeral datatype. Integral datatypes include - TINYINT, SMALLINT, INT, BIGINT.
+ * Note that with integral data types only equals("=") and not equals("!=") operators
+ * are supported. There are no operator restrictions with String.
+ */
+ if (!colType.equalsIgnoreCase(serdeConstants.STRING_TYPE_NAME)
+ && (!isIntegralSupported || !serdeConstants.IntegralTypes.contains(colType))) {
--- End diff --
Should the above be an OR condition?
---
[GitHub] incubator-hawq issue #1287: HAWQ-1527 Enabled partition filtering for integr...
Posted by radarwave <gi...@git.apache.org>.
Github user radarwave commented on the issue:
https://github.com/apache/incubator-hawq/pull/1287
@outofmem0ry The PR is merged, please close this PR. Thanks.
---
[GitHub] incubator-hawq issue #1287: HAWQ-1527 Enabled partition filtering for integr...
Posted by outofmem0ry <gi...@git.apache.org>.
Github user outofmem0ry commented on the issue:
https://github.com/apache/incubator-hawq/pull/1287
Thanks @radarwave
---
[GitHub] incubator-hawq pull request #1287: HAWQ-1527 Enabled partition filtering for...
Posted by outofmem0ry <gi...@git.apache.org>.
Github user outofmem0ry commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1287#discussion_r141186627
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenter.java ---
@@ -418,8 +427,15 @@ private boolean buildSingleFilter(Object filter,
return false;
}
- if (!partitionkeyTypes.get(filterColumnName).equalsIgnoreCase(serdeConstants.STRING_TYPE_NAME)) {
- LOG.debug("Filter type is not string type , ignore this filter for hive: "
+ /*
+ * HAWQ-1527 - Filtering only supported for partition columns of type string or
+ * intgeral datatype. Integral datatypes include - TINYINT, SMALLINT, INT, BIGINT.
+ * Note that with integral data types only equals("=") and not equals("!=") operators
+ * are supported. There are no operator restrictions with String.
+ */
+ if (!colType.equalsIgnoreCase(serdeConstants.STRING_TYPE_NAME)
+ && (!isIntegralSupported || !serdeConstants.IntegralTypes.contains(colType))) {
--- End diff --
@lavjain - This takes care of data types that are neither integral nor string. For example, if the column's data type is date, we don't support filtering on any operator. It should return false. If this would have been an OR , pxf will construct a filter since `!colType.equalsIgnoreCase(serdeConstants.STRING_TYPE_NAME)` is true. This filter will then be passed to hive which will throw an error as filtering is not supported for date data when accessing hive through hcatalog.
Keeping this in mind, I have added a unit test case for the above scenario.
---