You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/04/08 02:58:14 UTC
[GitHub] [incubator-doris] wuyunfeng opened a new pull request #3277: [Doris
on es] Support compound_and predicate push down to Elasticsearch
wuyunfeng opened a new pull request #3277: [Doris on es] Support compound_and predicate push down to Elasticsearch
URL: https://github.com/apache/incubator-doris/pull/3277
Relate Issue: https://github.com/apache/incubator-doris/issues/3248
SQL:
```
select * from test where (k2 = 6 and k3 = 1) or (k2 = 2 and k3 =3 and k4 = 'beijing');
```
Output filter:
```
ConstantScore(((#k2:[6 TO 6] #k3:[7 TO 7]) (#(#k2:[2 TO 2] #k3:[3 TO 3]) #k4:beijing))~1))^0.0
```
SQL:
```
select * from test where (k2 = 6 or k3 = 7) or (k2 = 2 and k3 =3 and (k4 = 'beijing' or k4 = 'zhaochun'));
```
Output filter:
```
(k2:[6 TO 6] k3:[7 TO 7] (#(#k2:[2 TO 2] #k3:[3 TO 3]) #((k4:beijing k4:zhaochun)~1)))~1
```
SQL:
```
select * from test where (k2 = 6 or k3 = 7) or (k2 = 2 and abs(k3) =3 and (k4 = 'beijing' or k4 = 'zhaochun'));
```
Output filter (`abs` can not be pushed down to es, so doris on es would not process this scenario ):
```
match_all
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #3277: [Doris on es]
Support compound_and predicate push down to Elasticsearch
Posted by GitBox <gi...@apache.org>.
imay merged pull request #3277: [Doris on es] Support compound_and predicate push down to Elasticsearch
URL: https://github.com/apache/incubator-doris/pull/3277
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3277:
[Doris on es] Support compound_and predicate push down to Elasticsearch
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3277: [Doris on es] Support compound_and predicate push down to Elasticsearch
URL: https://github.com/apache/incubator-doris/pull/3277#discussion_r405246814
##########
File path: be/src/exec/es/es_predicate.cpp
##########
@@ -384,10 +390,27 @@ Status EsPredicate::build_disjuncts_list(const Expr* conjunct) {
_disjuncts.push_back(predicate);
return Status::OK();
- }
-
+ }
if (TExprNodeType::COMPOUND_PRED == conjunct->node_type()) {
if (TExprOpcode::COMPOUND_OR != conjunct->op()) {
+ // processe COMPOUND_AND, such as:
+ // k = 1 or (k1 = 7 and (k2 in (6,7) or k3 = 12))
+ // k1 = 7 and (k2 in (6,7) or k3 = 12) is compound pred, we should rebuild this sub tree
+ if (TExprOpcode::COMPOUND_AND == conjunct->op()) {
Review comment:
```suggestion
if (conjunct->op() == TExprOpcode::COMPOUND_AND) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng commented on issue #3277: [Doris on es]
Support compound_and predicate push down to Elasticsearch
Posted by GitBox <gi...@apache.org>.
wuyunfeng commented on issue #3277: [Doris on es] Support compound_and predicate push down to Elasticsearch
URL: https://github.com/apache/incubator-doris/pull/3277#issuecomment-610948248
https://github.com/apache/incubator-doris/issues/3249
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3277:
[Doris on es] Support compound_and predicate push down to Elasticsearch
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3277: [Doris on es] Support compound_and predicate push down to Elasticsearch
URL: https://github.com/apache/incubator-doris/pull/3277#discussion_r405250902
##########
File path: be/src/exec/es/es_predicate.cpp
##########
@@ -384,10 +390,27 @@ Status EsPredicate::build_disjuncts_list(const Expr* conjunct) {
_disjuncts.push_back(predicate);
return Status::OK();
- }
-
+ }
if (TExprNodeType::COMPOUND_PRED == conjunct->node_type()) {
if (TExprOpcode::COMPOUND_OR != conjunct->op()) {
+ // processe COMPOUND_AND, such as:
+ // k = 1 or (k1 = 7 and (k2 in (6,7) or k3 = 12))
+ // k1 = 7 and (k2 in (6,7) or k3 = 12) is compound pred, we should rebuild this sub tree
+ if (TExprOpcode::COMPOUND_AND == conjunct->op()) {
Review comment:
There are three possibilities here. AND, OR, NOT, For me, `NOT` can be supported too.
And you can do as follow
```
if (conjunct->op() == AND)
else if (conjunct->op() == NOT)
DCHECK(op() == OR)
// handle OR
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng commented on issue #3277: [Doris on es]
Support compound_and predicate push down to Elasticsearch
Posted by GitBox <gi...@apache.org>.
wuyunfeng commented on issue #3277: [Doris on es] Support compound_and predicate push down to Elasticsearch
URL: https://github.com/apache/incubator-doris/pull/3277#issuecomment-610900391
@imay OK. please check again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3277:
[Doris on es] Support compound_and predicate push down to Elasticsearch
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3277: [Doris on es] Support compound_and predicate push down to Elasticsearch
URL: https://github.com/apache/incubator-doris/pull/3277#discussion_r405249722
##########
File path: be/src/exec/es/es_query_builder.cpp
##########
@@ -359,6 +370,17 @@ void BooleanQueryBuilder::validate(const std::vector<EsPredicate*>& espredicates
}
break;
}
+ case TExprNodeType::COMPOUND_PRED: {
+ ExtCompAndPredicates* compound_predicates = (ExtCompAndPredicates *)predicate;
+ std::vector<bool> list;
+ validate(compound_predicates->conjuncts, &list);
+ for(int i = list.size() - 1; i >= 0; i--) {
+ if(!list[i]) {
+ flag = false;
+ break;
+ }
+ }
Review comment:
better to add a `break` clause here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3277:
[Doris on es] Support compound_and predicate push down to Elasticsearch
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3277: [Doris on es] Support compound_and predicate push down to Elasticsearch
URL: https://github.com/apache/incubator-doris/pull/3277#discussion_r405245915
##########
File path: be/src/exec/es/es_predicate.h
##########
@@ -85,6 +86,15 @@ struct ExtPredicate {
TExprNodeType::type node_type;
};
+// this used for placeholder for compound_and
+struct ExtCompAndPredicates : public ExtPredicate {
+ ExtCompAndPredicates(std::vector<EsPredicate*> conjuncts) : ExtPredicate(TExprNodeType::COMPOUND_PRED), op(TExprOpcode::COMPOUND_AND), conjuncts(conjuncts) {
Review comment:
```suggestion
ExtCompAndPredicates(const std::vector<EsPredicate*>& conjuncts) : ExtPredicate(TExprNodeType::COMPOUND_PRED), op(TExprOpcode::COMPOUND_AND), conjuncts(conjuncts) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org