You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by jiadexin <gi...@git.apache.org> on 2016/08/05 07:59:15 UTC
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
GitHub user jiadexin opened a pull request:
https://github.com/apache/incubator-hawq/pull/837
HAWQ-779 support pxf filter pushdwon at the 'CREATE PLAN' stage ,and \u2026
1.support pxf filter pushdwon at the 'CREATE PLAN' stage -- src/backend/optimizer/plan/createplan.c
2.Due to '1' causes produce HAWQ-953 error, modify HiveDataFragmenter.java
3.add 'Date type' filter and 'HDOP_LIKE' op -- pxffilters.h,pxffilters.c,FilterParser.java , and update the corresponding test -- FilterParserTest.java,pxffilters_test.c
4.Due to '2' cause,modified HBaseFilterBuilder.java to handle 'HDOP_LIKE' op.
5. 'Date filter' and 'HDOP_LIKE' used in pxf-solr/pxf-jdbc(https://github.com/inspur-insight/pxf-plugin)
6.By this amendment, I think: 'PXF Filter' architecture coupling is too high, I just want to add new types and op, but had to modify other components .i hope to improve the architecture .
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/inspur-insight/incubator-hawq HAWQ-779
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/837.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #837
----
commit 3518b4af22af281909140bc011884420de540cc6
Author: Devin Jia <ji...@inspur.com>
Date: 2016-08-05T07:05:50Z
HAWQ-779 support pxf filter pushdwon at the 'CREATE PLAN' stage ,and more filter type & op
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #837: HAWQ-779 support pxf filter pushdwon at the 'CREA...
Posted by vraghavan78 <gi...@git.apache.org>.
Github user vraghavan78 commented on the issue:
https://github.com/apache/incubator-hawq/pull/837
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by jiadexin <gi...@git.apache.org>.
Github user jiadexin commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r79295630
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1144,9 +1144,15 @@ static char** create_pxf_plan(char **segdb_file_map, RelOptInfo *rel, int total_
Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, ctx->root)->relid);
- segdb_work_map = map_hddata_2gp_segments(uri_str,
+ if (pxf_enable_filter_pushdown){
--- End diff --
you are right.
My C language unskilled, so...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by jiadexin <gi...@git.apache.org>.
Github user jiadexin commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r74380940
--- Diff: src/backend/access/external/test/pxffilters_test.c ---
@@ -61,7 +62,7 @@ test__supported_filter_type(void **state)
/* go over pxf_supported_types array */
int nargs = sizeof(pxf_supported_types) / sizeof(Oid);
- assert_int_equal(nargs, 12);
+ assert_int_equal(nargs, 13);
--- End diff --
This test is to check pxf_supported_types number, it's old value is hard-coded too(be 12), after an increase of DATEOID become 13.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by jiadexin <gi...@git.apache.org>.
Github user jiadexin closed the pull request at:
https://github.com/apache/incubator-hawq/pull/837
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by kavinderd <gi...@git.apache.org>.
Github user kavinderd commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r76098154
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1146,7 +1146,7 @@ static char** create_pxf_plan(char **segdb_file_map, RelOptInfo *rel, int total_
Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, ctx->root)->relid);
segdb_work_map = map_hddata_2gp_segments(uri_str,
total_segs, segs_participating,
- relation, NULL);
+ relation, ctx->root->parse->jointree->quals);
--- End diff --
Can you ensure that the filters are only passed when the guc `pxf_enable_filter_pushdown` is enabled. Such as is done here: https://github.com/apache/incubator-hawq/blob/316c5048f303513c40e8194b15024df12a3d76ad/src/backend/access/external/pxffilters.c#L554
Currently if this guc is disabled filters should not be passed to PXF
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by GodenYao <gi...@git.apache.org>.
Github user GodenYao commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r76469613
--- Diff: pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java ---
@@ -165,6 +165,14 @@ private Filter handleSimpleOperations(FilterParser.Operation opId,
ByteArrayComparable comparator = getComparator(hbaseColumn.columnTypeCode(),
constant.constant());
+ if(operatorsMap.get(opId) == null){
+ //HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --
comparator
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by hsyuan <gi...@git.apache.org>.
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r76370526
--- Diff: pxf/pxf-api/src/test/java/org/apache/hawq/pxf/api/FilterParserTest.java ---
@@ -215,6 +215,10 @@ public void parseColumnOnLeft() throws Exception {
filter = "a1c2o7";
op = Operation.HDOP_AND;
runParseOneOperation("this filter was build from HDOP_AND", filter, op);
+
+ filter = "a1c2o8";
+ op = Operation.HDOP_LIKE;
+ runParseOneOperation("this filter was build from HDOP_LIKE", filter, op);
--- End diff --
build -> built
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by jiadexin <gi...@git.apache.org>.
Github user jiadexin commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r76547017
--- Diff: pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java ---
@@ -165,6 +165,14 @@ private Filter handleSimpleOperations(FilterParser.Operation opId,
ByteArrayComparable comparator = getComparator(hbaseColumn.columnTypeCode(),
constant.constant());
+ if(operatorsMap.get(opId) == null){
+ //HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --
should i develop a HBase LIKE Filter ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r78508860
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1144,9 +1144,15 @@ static char** create_pxf_plan(char **segdb_file_map, RelOptInfo *rel, int total_
Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, ctx->root)->relid);
- segdb_work_map = map_hddata_2gp_segments(uri_str,
+ if (pxf_enable_filter_pushdown){
--- End diff --
can be refactored:
```
map_hddata_2gp_segments(uri_str,
total_segs, segs_participating,
relation,
pxf_enable_filter_pushdown ? ctx->root->parse->jointree->quals : NULL);
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by kavinderd <gi...@git.apache.org>.
Github user kavinderd commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r74286229
--- Diff: src/backend/access/external/test/pxffilters_test.c ---
@@ -61,7 +62,7 @@ test__supported_filter_type(void **state)
/* go over pxf_supported_types array */
int nargs = sizeof(pxf_supported_types) / sizeof(Oid);
- assert_int_equal(nargs, 12);
+ assert_int_equal(nargs, 13);
--- End diff --
Maybe have this value derived instead of hard-coded number
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by jiadexin <gi...@git.apache.org>.
Github user jiadexin commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r76368428
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1146,7 +1146,7 @@ static char** create_pxf_plan(char **segdb_file_map, RelOptInfo *rel, int total_
Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, ctx->root)->relid);
segdb_work_map = map_hddata_2gp_segments(uri_str,
total_segs, segs_participating,
- relation, NULL);
+ relation, ctx->root->parse->jointree->quals);
--- End diff --
ok.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by hsyuan <gi...@git.apache.org>.
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r76479540
--- Diff: pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java ---
@@ -165,6 +165,14 @@ private Filter handleSimpleOperations(FilterParser.Operation opId,
ByteArrayComparable comparator = getComparator(hbaseColumn.columnTypeCode(),
constant.constant());
+ if(operatorsMap.get(opId) == null){
+ //HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --
HBase does not support
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by jiadexin <gi...@git.apache.org>.
Github user jiadexin commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r79295676
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1144,9 +1144,15 @@ static char** create_pxf_plan(char **segdb_file_map, RelOptInfo *rel, int total_
Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, ctx->root)->relid);
- segdb_work_map = map_hddata_2gp_segments(uri_str,
+ if (pxf_enable_filter_pushdown){
--- End diff --
you are right.
my C language unskilled, so...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by kavinderd <gi...@git.apache.org>.
Github user kavinderd commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r77047594
--- Diff: pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java ---
@@ -165,6 +165,14 @@ private Filter handleSimpleOperations(FilterParser.Operation opId,
ByteArrayComparable comparator = getComparator(hbaseColumn.columnTypeCode(),
constant.constant());
+ if(operatorsMap.get(opId) == null){
+ //HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --
No, @hsyuan comment was that the comment should read "//HBase does not support..." HBase does not support the LIKE Filter as far as I am aware
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #837: HAWQ-779 support pxf filter pushdwon at the 'CREA...
Posted by GodenYao <gi...@git.apache.org>.
Github user GodenYao commented on the issue:
https://github.com/apache/incubator-hawq/pull/837
merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Posted by jiadexin <gi...@git.apache.org>.
Github user jiadexin commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r76546715
--- Diff: pxf/pxf-api/src/test/java/org/apache/hawq/pxf/api/FilterParserTest.java ---
@@ -215,6 +215,10 @@ public void parseColumnOnLeft() throws Exception {
filter = "a1c2o7";
op = Operation.HDOP_AND;
runParseOneOperation("this filter was build from HDOP_AND", filter, op);
+
+ filter = "a1c2o8";
+ op = Operation.HDOP_LIKE;
+ runParseOneOperation("this filter was build from HDOP_LIKE", filter, op);
--- End diff --
This reference to the previous code _`runParseOneOperation("this filter was build from HDOP_AND", filter, op)`_ , has been corrected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---