You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2022/07/02 15:12:27 UTC
[doris] branch dev-1.0.1 updated (94a883d68e -> 70354f2f4d)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/doris.git
from 94a883d68e [hot-fix] fix a typo error and limit the max wait time in VOlapTableSink::send (#10552)
new 378a5a3a4e [fix](proc) Fix show proc '/current_query_stmts' error due to wrong index for execTime (#10488)
new dcb66e084f [bugfix]fix core dump on outfile with expr (#10491)
new 70354f2f4d [fix](planner)infer predicate generate infered predicate using wrong information from another scope (#10519)
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/vec/runtime/vfile_result_writer.cpp | 34 +++++---
be/src/vec/runtime/vfile_result_writer.h | 14 ++--
be/src/vec/sink/vresult_file_sink.cpp | 13 ++--
be/src/vec/sink/vresult_file_sink.h | 18 ++---
.../java/org/apache/doris/analysis/SlotRef.java | 2 +-
.../proc/CurrentQueryStatementsProcNode.java | 2 +-
.../org/apache/doris/common/proc/TrashProcDir.java | 4 +-
.../org/apache/doris/rewrite/InferFiltersRule.java | 90 ++++++++++------------
.../apache/doris/rewrite/InferFiltersRuleTest.java | 70 +++++++++++++----
9 files changed, 147 insertions(+), 100 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 01/03: [fix](proc) Fix show proc '/current_query_stmts' error due to wrong index for execTime (#10488)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 378a5a3a4e14b3280d37c2092a8fb7c83709c3d3
Author: caiconghui <55...@users.noreply.github.com>
AuthorDate: Wed Jun 29 17:41:47 2022 +0800
[fix](proc) Fix show proc '/current_query_stmts' error due to wrong index for execTime (#10488)
Co-authored-by: caiconghui1 <ca...@jd.com>
---
.../org/apache/doris/common/proc/CurrentQueryStatementsProcNode.java | 2 +-
.../src/main/java/org/apache/doris/common/proc/TrashProcDir.java | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryStatementsProcNode.java b/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryStatementsProcNode.java
index 3b0474d9b0..e9ccb9c3e3 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryStatementsProcNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/proc/CurrentQueryStatementsProcNode.java
@@ -35,7 +35,7 @@ public class CurrentQueryStatementsProcNode implements ProcNodeInterface {
.add("QueryId").add("ConnectionId").add("Database").add("User")
.add("ExecTime").add("SqlHash").add("Statement").build();
- private static final int EXEC_TIME_INDEX = 5;
+ private static final int EXEC_TIME_INDEX = 4;
@Override
public ProcResult fetchResult() throws AnalysisException {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/proc/TrashProcDir.java b/fe/fe-core/src/main/java/org/apache/doris/common/proc/TrashProcDir.java
index 493ce936cc..b54f6f8aa9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/proc/TrashProcDir.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/proc/TrashProcDir.java
@@ -94,9 +94,9 @@ public class TrashProcDir implements ProcDirInterface {
}
}
- List<String> backendInfo = new ArrayList<String>();
+ List<String> backendInfo = new ArrayList<>();
backendInfo.add(String.valueOf(backend.getId()));
- backendInfo.add(backend.getHost() + ":" + String.valueOf(backend.getHeartbeatPort()));
+ backendInfo.add(backend.getHost() + ":" + backend.getHeartbeatPort());
if (trashUsedCapacityB != null) {
Pair<Double, String> trashUsedCapacity = DebugUtil.getByteUint(trashUsedCapacityB);
backendInfo.add(DebugUtil.DECIMAL_FORMAT_SCALE_3.format(trashUsedCapacity.first) + " "
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 03/03: [fix](planner)infer predicate generate infered predicate using wrong information from another scope (#10519)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 70354f2f4d4edce7a0d624bd0eb4b90dfbecdf44
Author: morrySnow <10...@users.noreply.github.com>
AuthorDate: Sat Jul 2 22:41:04 2022 +0800
[fix](planner)infer predicate generate infered predicate using wrong information from another scope (#10519)
This PR fix a bug in predicate inference.
The original predicate inference compare two slot without SlotId. This will arise an error when a query has SetOperand and more than one SetOperand's child use same table alias. e.g.
```
select * from tb1 inner join tb2 on tb1.k1 = tb2.k1
union
select * from tb1 inner join tb2 on tb1.k2 = tb2.k2 where tb1.k1 = 3;
```
in this case, we infer a predicate `tb2.k1 = 3` on table 'tbl2' of SetOperand's second child by mistake.
---
.../java/org/apache/doris/analysis/SlotRef.java | 2 +-
.../org/apache/doris/rewrite/InferFiltersRule.java | 90 ++++++++++------------
.../apache/doris/rewrite/InferFiltersRuleTest.java | 70 +++++++++++++----
3 files changed, 99 insertions(+), 63 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java
index f2f87e0586..09c6138a6d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java
@@ -303,7 +303,7 @@ public class SlotRef extends Expr {
if ((col == null) != (other.col == null)) {
return false;
}
- if (col != null && !col.toLowerCase().equals(other.col.toLowerCase())) {
+ if (col != null && !col.equalsIgnoreCase(other.col)) {
return false;
}
return true;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/rewrite/InferFiltersRule.java b/fe/fe-core/src/main/java/org/apache/doris/rewrite/InferFiltersRule.java
index 47f8153485..c83d3b94c8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/rewrite/InferFiltersRule.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/rewrite/InferFiltersRule.java
@@ -44,6 +44,8 @@ import java.util.Map;
/**
* The function of this rule is to derive a new predicate based on the current predicate.
+ *
+ * <pre>
* eg.
* t1.id = t2.id and t2.id = t3.id and t3.id = 100;
* -->
@@ -52,8 +54,9 @@ import java.util.Map;
* 1. Register a new rule InferFiltersRule and add it to GlobalState.
* 2. Traverse Conjunct to construct on/where equivalence connection, numerical connection and isNullPredicate.
* 3. Use Warshall to infer all equivalence connections.
- * details:https://en.wikipedia.org/wiki/Floyd%E2%80%93Warshall_algorithm
+ * details: <a href="url">https://en.wikipedia.org/wiki/Floyd%E2%80%93Warshall_algorithm</a>
* 4. Construct additional numerical connections and isNullPredicate.
+ * </pre>
*/
public class InferFiltersRule implements ExprRewriteRule {
private final static Logger LOG = LogManager.getLogger(InferFiltersRule.class);
@@ -127,10 +130,10 @@ public class InferFiltersRule implements ExprRewriteRule {
if (!newExprWithState.isEmpty()) {
Expr rewriteExpr = expr;
- for (int index = 0; index < newExprWithState.size(); index++) {
- if (newExprWithState.get(index).second) {
- rewriteExpr = new CompoundPredicate(CompoundPredicate.Operator.AND,
- rewriteExpr, newExprWithState.get(index).first);
+ for (Pair<Expr, Boolean> exprBooleanPair : newExprWithState) {
+ if (exprBooleanPair.second) {
+ rewriteExpr = new CompoundPredicate(CompoundPredicate.Operator.AND, rewriteExpr,
+ exprBooleanPair.first);
}
}
return rewriteExpr;
@@ -169,9 +172,9 @@ public class InferFiltersRule implements ExprRewriteRule {
}
if (conjunct instanceof BinaryPredicate
- && conjunct.getChild(0) != null
- && conjunct.getChild(1) != null) {
- if (conjunct.getChild(0).unwrapSlotRef() instanceof SlotRef
+ && conjunct.getChild(0) != null
+ && conjunct.getChild(1) != null) {
+ if (conjunct.getChild(0).unwrapSlotRef() != null
&& conjunct.getChild(1) instanceof LiteralExpr) {
Pair<Expr, Expr> pair = new Pair<>(conjunct.getChild(0).unwrapSlotRef(), conjunct.getChild(1));
if (!slotToLiteralDeDuplication.contains(pair)) {
@@ -184,8 +187,8 @@ public class InferFiltersRule implements ExprRewriteRule {
analyzer.registerGlobalSlotToLiteralDeDuplication(pair);
}
} else if (((BinaryPredicate) conjunct).getOp().isEquivalence()
- && conjunct.getChild(0).unwrapSlotRef() instanceof SlotRef
- && conjunct.getChild(1).unwrapSlotRef() instanceof SlotRef) {
+ && conjunct.getChild(0).unwrapSlotRef() != null
+ && conjunct.getChild(1).unwrapSlotRef() != null) {
Pair<Expr, Expr> pair = new Pair<>(conjunct.getChild(0).unwrapSlotRef(),
conjunct.getChild(1).unwrapSlotRef());
Pair<Expr, Expr> eqPair = new Pair<>(conjunct.getChild(1).unwrapSlotRef(),
@@ -202,7 +205,7 @@ public class InferFiltersRule implements ExprRewriteRule {
}
} else if (conjunct instanceof IsNullPredicate
&& conjunct.getChild(0) != null
- && conjunct.getChild(0).unwrapSlotRef() instanceof SlotRef) {
+ && conjunct.getChild(0).unwrapSlotRef() != null) {
if (!isNullDeDuplication.contains(conjunct.getChild(0).unwrapSlotRef())
&& ((IsNullPredicate) conjunct).isNotNull()) {
isNullDeDuplication.add(conjunct.getChild(0).unwrapSlotRef());
@@ -214,7 +217,7 @@ public class InferFiltersRule implements ExprRewriteRule {
}
} else if (conjunct instanceof InPredicate
&& conjunct.getChild(0) != null
- && conjunct.getChild(0).unwrapSlotRef() instanceof SlotRef) {
+ && conjunct.getChild(0).unwrapSlotRef() != null) {
if (!inDeDuplication.contains(conjunct.getChild(0).unwrapSlotRef())) {
inDeDuplication.add(conjunct.getChild(0).unwrapSlotRef());
inExpr.add(conjunct);
@@ -234,10 +237,10 @@ public class InferFiltersRule implements ExprRewriteRule {
* old expr:t1.id = t2.id and t2.id = t3.id and t3.id = t4.id
* new expr:t1.id = t2.id and t2.id = t3.id and t3.id = t4.id and t1.id = t3.id and t1.id = t4.id and t2.id = t4.id
*
- * @param slotEqSlotExpr
- * @param slotEqSlotDeDuplication
+ * @param slotEqSlotExpr slot to slot exprs
+ * @param slotEqSlotDeDuplication set pairs in slot = slot exprs
* @param exprToWarshallArraySubscript: A Map the key is Expr, the value is int
- * @param warshallArraySubscriptToExpr: A Map the key is int, the value is exper
+ * @param warshallArraySubscriptToExpr: A Map the key is int, the value is expr
*/
private void genNewSlotEqSlotPredicate(List<Expr> slotEqSlotExpr,
Set<Pair<Expr, Expr>> slotEqSlotDeDuplication,
@@ -267,9 +270,9 @@ public class InferFiltersRule implements ExprRewriteRule {
*
* @param warshall: Two-dimensional array
* @param arrayMaxSize: slotEqSlotExpr.size() * 2
- * @param slotEqSlotExpr
- * @param exprToWarshallArraySubscript
- * @param warshallArraySubscriptToExpr
+ * @param slotEqSlotExpr slot to slot exprs
+ * @param exprToWarshallArraySubscript expr to offset in Warshall array
+ * @param warshallArraySubscriptToExpr offset in Warshall array to expr
* @return needGenWarshallArray. True:needGen; False:don't needGen
*/
private boolean initWarshallArray(int warshall[][],
@@ -280,8 +283,8 @@ public class InferFiltersRule implements ExprRewriteRule {
boolean needGenWarshallArray = false;
int index = 0;
for (Expr slotEqSlot : slotEqSlotExpr) {
- int row = 0;
- int column = 0;
+ int row;
+ int column;
if (!exprToWarshallArraySubscript.containsKey(slotEqSlot.getChild(0))) {
exprToWarshallArraySubscript.put(slotEqSlot.getChild(0), index);
warshallArraySubscriptToExpr.put(index, slotEqSlot.getChild(0));
@@ -302,7 +305,7 @@ public class InferFiltersRule implements ExprRewriteRule {
if (row >= arrayMaxSize
|| column >= arrayMaxSize) {
- LOG.debug("Error row or column", row, column, arrayMaxSize);
+ LOG.debug("Error row {} or column {}, but max size is {}.", row, column, arrayMaxSize);
needGenWarshallArray = false;
break;
} else {
@@ -394,19 +397,18 @@ public class InferFiltersRule implements ExprRewriteRule {
Analyzer analyzer,
ExprRewriter.ClauseType clauseType) {
SlotRef checkSlot = slotToLiteral.getChild(0).unwrapSlotRef();
- if (checkSlot instanceof SlotRef) {
+ if (checkSlot != null) {
for (Expr conjunct : slotEqSlotExpr) {
SlotRef leftSlot = conjunct.getChild(0).unwrapSlotRef();
SlotRef rightSlot = conjunct.getChild(1).unwrapSlotRef();
- if (leftSlot instanceof SlotRef
- && rightSlot instanceof SlotRef) {
- if (checkSlot.notCheckDescIdEquals(leftSlot)) {
+ if (leftSlot != null && rightSlot != null) {
+ if (checkSlot.equals(leftSlot)) {
addNewBinaryPredicate(genNewBinaryPredicate(slotToLiteral, rightSlot),
slotToLiteralDeDuplication, newExprWithState,
isNeedInfer(rightSlot, leftSlot, analyzer, clauseType),
analyzer, clauseType);
- } else if (checkSlot.notCheckDescIdEquals(rightSlot)) {
+ } else if (checkSlot.equals(rightSlot)) {
addNewBinaryPredicate(genNewBinaryPredicate(slotToLiteral, leftSlot),
slotToLiteralDeDuplication, newExprWithState,
isNeedInfer(leftSlot, rightSlot, analyzer, clauseType),
@@ -429,17 +431,15 @@ public class InferFiltersRule implements ExprRewriteRule {
boolean ret = false;
TupleId newTid = newSlot.getDesc().getParent().getRef().getId();
TupleId checkTid = checkSlot.getDesc().getParent().getRef().getId();
- boolean needChange = false;
Pair<TupleId, TupleId> tids = new Pair<>(newTid, checkTid);
if (analyzer.isContainTupleIds(tids)) {
JoinOperator joinOperator = analyzer.getAnyTwoTablesJoinOp(tids);
- ret = checkNeedInfer(joinOperator, needChange, clauseType);
+ ret = checkNeedInfer(joinOperator, false, clauseType);
} else {
Pair<TupleId, TupleId> changeTids = new Pair<>(checkTid, newTid);
if (analyzer.isContainTupleIds(changeTids)) {
- needChange = true;
JoinOperator joinOperator = analyzer.getAnyTwoTablesJoinOp(changeTids);
- ret = checkNeedInfer(joinOperator, needChange, clauseType);
+ ret = checkNeedInfer(joinOperator, true, clauseType);
}
}
return ret;
@@ -475,8 +475,7 @@ public class InferFiltersRule implements ExprRewriteRule {
private Expr genNewBinaryPredicate(Expr oldExpr, Expr newSlot) {
if (oldExpr instanceof BinaryPredicate) {
BinaryPredicate oldBP = (BinaryPredicate) oldExpr;
- BinaryPredicate newBP = new BinaryPredicate(oldBP.getOp(), newSlot, oldBP.getChild(1));
- return newBP;
+ return new BinaryPredicate(oldBP.getOp(), newSlot, oldBP.getChild(1));
}
return oldExpr;
}
@@ -535,17 +534,16 @@ public class InferFiltersRule implements ExprRewriteRule {
if (expr instanceof IsNullPredicate) {
IsNullPredicate isNullPredicate = (IsNullPredicate)expr;
SlotRef checkSlot = isNullPredicate.getChild(0).unwrapSlotRef();
- if (checkSlot instanceof SlotRef) {
+ if (checkSlot != null) {
for (Expr conjunct : slotEqSlotExpr) {
SlotRef leftSlot = conjunct.getChild(0).unwrapSlotRef();
SlotRef rightSlot = conjunct.getChild(1).unwrapSlotRef();
- if (leftSlot instanceof SlotRef
- && rightSlot instanceof SlotRef) {
- if (checkSlot.notCheckDescIdEquals(leftSlot) && isNullPredicate.isNotNull()) {
+ if (leftSlot != null && rightSlot != null) {
+ if (checkSlot.equals(leftSlot) && isNullPredicate.isNotNull()) {
addNewIsNotNullPredicate(genNewIsNotNullPredicate(isNullPredicate, rightSlot),
isNullDeDuplication, newExprWithState, analyzer, clauseType);
- } else if (checkSlot.notCheckDescIdEquals(rightSlot)) {
+ } else if (checkSlot.equals(rightSlot)) {
addNewIsNotNullPredicate(genNewIsNotNullPredicate(isNullPredicate, leftSlot),
isNullDeDuplication, newExprWithState, analyzer, clauseType);
}
@@ -560,11 +558,7 @@ public class InferFiltersRule implements ExprRewriteRule {
* @return new IsNullPredicate.
*/
private Expr genNewIsNotNullPredicate(IsNullPredicate oldExpr, Expr newSlot) {
- if (oldExpr instanceof IsNullPredicate) {
- IsNullPredicate newExpr = new IsNullPredicate(newSlot, oldExpr.isNotNull());
- return newExpr;
- }
- return oldExpr;
+ return oldExpr != null ? new IsNullPredicate(newSlot, oldExpr.isNotNull()) : null;
}
/**
@@ -616,19 +610,18 @@ public class InferFiltersRule implements ExprRewriteRule {
if (inExpr instanceof InPredicate) {
InPredicate inpredicate = (InPredicate) inExpr;
SlotRef checkSlot = inpredicate.getChild(0).unwrapSlotRef();
- if (checkSlot instanceof SlotRef) {
+ if (checkSlot != null) {
for (Expr conjunct : slotEqSlotExpr) {
SlotRef leftSlot = conjunct.getChild(0).unwrapSlotRef();
SlotRef rightSlot = conjunct.getChild(1).unwrapSlotRef();
- if (leftSlot instanceof SlotRef
- && rightSlot instanceof SlotRef) {
- if (checkSlot.notCheckDescIdEquals(leftSlot)) {
+ if (leftSlot != null && rightSlot != null) {
+ if (checkSlot.equals(leftSlot)) {
addNewInPredicate(genNewInPredicate(inpredicate, rightSlot),
inDeDuplication, newExprWithState,
isNeedInfer(rightSlot, leftSlot, analyzer, clauseType),
analyzer, clauseType);
- } else if (checkSlot.notCheckDescIdEquals(rightSlot)) {
+ } else if (checkSlot.equals(rightSlot)) {
addNewInPredicate(genNewInPredicate(inpredicate, leftSlot),
inDeDuplication, newExprWithState,
isNeedInfer(leftSlot, rightSlot, analyzer, clauseType),
@@ -647,8 +640,7 @@ public class InferFiltersRule implements ExprRewriteRule {
private Expr genNewInPredicate(Expr oldExpr, Expr newSlot) {
if (oldExpr instanceof InPredicate) {
InPredicate oldBP = (InPredicate) oldExpr;
- InPredicate newBP = new InPredicate(newSlot, oldBP.getListChildren(), oldBP.isNotIn());
- return newBP;
+ return new InPredicate(newSlot, oldBP.getListChildren(), oldBP.isNotIn());
}
return oldExpr;
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/rewrite/InferFiltersRuleTest.java b/fe/fe-core/src/test/java/org/apache/doris/rewrite/InferFiltersRuleTest.java
index 1ceb7f63d9..c5c5be25a6 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/rewrite/InferFiltersRuleTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/rewrite/InferFiltersRuleTest.java
@@ -33,8 +33,8 @@ import org.junit.Test;
import java.util.UUID;
public class InferFiltersRuleTest {
- private static String baseDir = "fe";
- private static String runningDir = baseDir + "/mocked/InferFiltersRuleTest/"
+ private static final String baseDir = "fe";
+ private static final String runningDir = baseDir + "/mocked/InferFiltersRuleTest/"
+ UUID.randomUUID() + "/";
private static DorisAssert dorisAssert;
private static final String DB_NAME = "db1";
@@ -58,7 +58,8 @@ public class InferFiltersRuleTest {
+ "distributed by hash(k1) buckets 3 properties('replication_num' = '1');";
dorisAssert.withTable(createTableSQL);
createTableSQL = "create table " + DB_NAME + "." + TABLE_NAME_3
- + " (k1 tinyint, k2 smallint, k3 int, k4 bigint, k5 largeint, k6 date, k7 datetime, k8 float, k9 double) "
+ + " (k1 tinyint, k2 smallint, k3 int, k4 bigint,"
+ + " k5 largeint, k6 date, k7 datetime, k8 float, k9 double) "
+ "distributed by hash(k1) buckets 3 properties('replication_num' = '1');";
dorisAssert.withTable(createTableSQL);
}
@@ -104,7 +105,8 @@ public class InferFiltersRuleTest {
SessionVariable sessionVariable = dorisAssert.getSessionVariable();
sessionVariable.setEnableInferPredicate(true);
Assert.assertTrue(sessionVariable.isEnableInferPredicate());
- String query = "select * from tb1 inner join tb2 inner join tb3 where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb3.k1 = 1";
+ String query = "select * from tb1 inner join tb2 inner join tb3"
+ + " where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb3.k1 = 1";
String planString = dorisAssert.query(query).explainQuery();
Assert.assertTrue(planString.contains("`tb2`.`k1` = 1"));
Assert.assertTrue(planString.contains("`tb1`.`k1` = 1"));
@@ -154,7 +156,8 @@ public class InferFiltersRuleTest {
SessionVariable sessionVariable = dorisAssert.getSessionVariable();
sessionVariable.setEnableInferPredicate(true);
Assert.assertTrue(sessionVariable.isEnableInferPredicate());
- String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1 right outer join tb3 on tb2.k1 = tb3.k1 and tb2.k1 = 1";
+ String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1"
+ + " right outer join tb3 on tb2.k1 = tb3.k1 and tb2.k1 = 1";
String planString = dorisAssert.query(query).explainQuery();
Assert.assertTrue(planString.contains("`tb1`.`k1` = 1"));
Assert.assertFalse(planString.contains("`tb3`.`k1` = 1"));
@@ -165,7 +168,8 @@ public class InferFiltersRuleTest {
SessionVariable sessionVariable = dorisAssert.getSessionVariable();
sessionVariable.setEnableInferPredicate(true);
Assert.assertTrue(sessionVariable.isEnableInferPredicate());
- String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1 right outer join tb3 on tb2.k1 = tb3.k1 and tb3.k1 = 1";
+ String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1"
+ + " right outer join tb3 on tb2.k1 = tb3.k1 and tb3.k1 = 1";
String planString = dorisAssert.query(query).explainQuery();
Assert.assertTrue(planString.contains("`tb2`.`k1` = 1"));
Assert.assertTrue(planString.contains("`tb1`.`k1` = 1"));
@@ -176,7 +180,8 @@ public class InferFiltersRuleTest {
SessionVariable sessionVariable = dorisAssert.getSessionVariable();
sessionVariable.setEnableInferPredicate(true);
Assert.assertTrue(sessionVariable.isEnableInferPredicate());
- String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1 right outer join tb3 on tb2.k1 = tb3.k1 where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb2.k1 = 1";
+ String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1 right outer join tb3 on tb2.k1 = tb3.k1"
+ + " where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb2.k1 = 1";
String planString = dorisAssert.query(query).explainQuery();
Assert.assertTrue(planString.contains("`tb1`.`k1` = 1"));
Assert.assertTrue(planString.contains("`tb3`.`k1` = 1"));
@@ -219,7 +224,8 @@ public class InferFiltersRuleTest {
SessionVariable sessionVariable = dorisAssert.getSessionVariable();
sessionVariable.setEnableInferPredicate(true);
Assert.assertTrue(sessionVariable.isEnableInferPredicate());
- String query = "select * from tb1 inner join tb2 inner join tb3 where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb3.k1 = 1";
+ String query = "select * from tb1 inner join tb2 inner join tb3"
+ + " where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb3.k1 = 1";
String planString = dorisAssert.query(query).explainQuery();
Assert.assertTrue(planString.contains("`tb2`.`k1` = 1"));
Assert.assertTrue(planString.contains("`tb1`.`k1` = 1"));
@@ -230,7 +236,8 @@ public class InferFiltersRuleTest {
SessionVariable sessionVariable = dorisAssert.getSessionVariable();
sessionVariable.setEnableInferPredicate(true);
Assert.assertTrue(sessionVariable.isEnableInferPredicate());
- String query = "select * from tb1 inner join tb2 left outer join tb3 on tb3.k1 = tb2.k1 where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb3.k1 = 1";
+ String query = "select * from tb1 inner join tb2 left outer join tb3 on tb3.k1 = tb2.k1"
+ + " where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb3.k1 = 1";
String planString = dorisAssert.query(query).explainQuery();
Assert.assertTrue(planString.contains("`tb2`.`k1` = 1"));
Assert.assertTrue(planString.contains("`tb1`.`k1` = 1"));
@@ -241,7 +248,8 @@ public class InferFiltersRuleTest {
SessionVariable sessionVariable = dorisAssert.getSessionVariable();
sessionVariable.setEnableInferPredicate(true);
Assert.assertTrue(sessionVariable.isEnableInferPredicate());
- String query = "select * from tb1 inner join tb2 left outer join tb3 on tb3.k1 = tb2.k1 where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb2.k1 = 1";
+ String query = "select * from tb1 inner join tb2 left outer join tb3 on tb3.k1 = tb2.k1"
+ + " where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb2.k1 = 1";
String planString = dorisAssert.query(query).explainQuery();
Assert.assertTrue(planString.contains("`tb1`.`k1` = 1"));
Assert.assertFalse(planString.contains("`tb3`.`k1` = 1"));
@@ -252,7 +260,8 @@ public class InferFiltersRuleTest {
SessionVariable sessionVariable = dorisAssert.getSessionVariable();
sessionVariable.setEnableInferPredicate(true);
Assert.assertTrue(sessionVariable.isEnableInferPredicate());
- String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1 right outer join tb3 on tb2.k1 = tb3.k1 where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb2.k1 = 1";
+ String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1 right outer join tb3 on tb2.k1 = tb3.k1"
+ + " where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb2.k1 = 1";
String planString = dorisAssert.query(query).explainQuery();
Assert.assertTrue(planString.contains("`tb1`.`k1` = 1"));
Assert.assertTrue(planString.contains("`tb3`.`k1` = 1"));
@@ -263,7 +272,8 @@ public class InferFiltersRuleTest {
SessionVariable sessionVariable = dorisAssert.getSessionVariable();
sessionVariable.setEnableInferPredicate(true);
Assert.assertTrue(sessionVariable.isEnableInferPredicate());
- String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1 right outer join tb3 on tb2.k1 = tb3.k1 where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb3.k1 = 1";
+ String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1 right outer join tb3 on tb2.k1 = tb3.k1"
+ + " where tb1.k1 = tb2.k1 and tb2.k1 = tb3.k1 and tb3.k1 = 1";
String planString = dorisAssert.query(query).explainQuery();
Assert.assertFalse(planString.contains("`tb2`.`k1` = 1"));
Assert.assertFalse(planString.contains("`tb1`.`k1` = 1"));
@@ -274,7 +284,8 @@ public class InferFiltersRuleTest {
SessionVariable sessionVariable = dorisAssert.getSessionVariable();
sessionVariable.setEnableInferPredicate(true);
Assert.assertTrue(sessionVariable.isEnableInferPredicate());
- String query = "select * from tb1 inner join tb2 inner join tb3 where tb1.k1 = tb3.k1 and tb2.k1 = tb3.k1 and tb1.k1 is not null";
+ String query = "select * from tb1 inner join tb2 inner join tb3"
+ + " where tb1.k1 = tb3.k1 and tb2.k1 = tb3.k1 and tb1.k1 is not null";
String planString = dorisAssert.query(query).explainQuery();
Assert.assertTrue(planString.contains("`tb3`.`k1` IS NOT NULL"));
Assert.assertTrue(planString.contains("`tb2`.`k1` IS NOT NULL"));
@@ -340,4 +351,37 @@ public class InferFiltersRuleTest {
String planString = dorisAssert.query(query).explainQuery();
Assert.assertTrue(planString.contains("`tb2`.`k1` = 1"));
}
+
+ @Test
+ public void testSameAliasWithSlotEqualToLiteralInDifferentUnionChildren() throws Exception {
+ SessionVariable sessionVariable = dorisAssert.getSessionVariable();
+ sessionVariable.setEnableInferPredicate(true);
+ Assert.assertTrue(sessionVariable.isEnableInferPredicate());
+ String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1"
+ + " union select * from tb1 inner join tb2 on tb1.k2 = tb2.k2 where tb1.k1 = 3";
+ String planString = dorisAssert.query(query).explainQuery();
+ Assert.assertFalse(planString.contains("`tb2`.`k1` = 3"));
+ }
+
+ @Test
+ public void testSameAliasWithSlotInPredicateInDifferentUnionChildren() throws Exception {
+ SessionVariable sessionVariable = dorisAssert.getSessionVariable();
+ sessionVariable.setEnableInferPredicate(true);
+ Assert.assertTrue(sessionVariable.isEnableInferPredicate());
+ String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1"
+ + " union select * from tb1 inner join tb2 on tb1.k2 = tb2.k2 where tb1.k1 in (3, 4, 5)";
+ String planString = dorisAssert.query(query).explainQuery();
+ Assert.assertFalse(planString.contains("`tb2`.`k1` IN (3, 4, 5)"));
+ }
+
+ @Test
+ public void testSameAliasWithSlotIsNullInDifferentUnionChildren() throws Exception {
+ SessionVariable sessionVariable = dorisAssert.getSessionVariable();
+ sessionVariable.setEnableInferPredicate(true);
+ Assert.assertTrue(sessionVariable.isEnableInferPredicate());
+ String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1"
+ + " union select * from tb1 inner join tb2 on tb1.k2 = tb2.k2 where tb1.k1 is not null";
+ String planString = dorisAssert.query(query).explainQuery();
+ Assert.assertFalse(planString.contains("`tb2`.`k1` IS NOT NULL"));
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 02/03: [bugfix]fix core dump on outfile with expr (#10491)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit dcb66e084fa58a9c6ce797f961dfac2fbc430fc3
Author: Pxl <95...@qq.com>
AuthorDate: Wed Jun 29 20:38:49 2022 +0800
[bugfix]fix core dump on outfile with expr (#10491)
remove log
---
be/src/vec/runtime/vfile_result_writer.cpp | 34 +++++++++++++++++++-----------
be/src/vec/runtime/vfile_result_writer.h | 14 ++++++------
be/src/vec/sink/vresult_file_sink.cpp | 13 ++++++------
be/src/vec/sink/vresult_file_sink.h | 18 ++++++++--------
4 files changed, 45 insertions(+), 34 deletions(-)
diff --git a/be/src/vec/runtime/vfile_result_writer.cpp b/be/src/vec/runtime/vfile_result_writer.cpp
index 71a748e565..6d4ecb8db1 100644
--- a/be/src/vec/runtime/vfile_result_writer.cpp
+++ b/be/src/vec/runtime/vfile_result_writer.cpp
@@ -17,6 +17,7 @@
#include "vec/runtime/vfile_result_writer.h"
+#include "common/status.h"
#include "exprs/expr_context.h"
#include "gutil/strings/numbers.h"
#include "gutil/strings/substitute.h"
@@ -35,22 +36,23 @@
#include "util/mysql_global.h"
#include "util/mysql_row_buffer.h"
#include "vec/core/block.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vexpr_context.h"
namespace doris::vectorized {
const size_t VFileResultWriter::OUTSTREAM_BUFFER_SIZE_BYTES = 1024 * 1024;
using doris::operator<<;
-VFileResultWriter::VFileResultWriter(const ResultFileOptions* file_opts,
- const TStorageBackendType::type storage_type,
- const TUniqueId fragment_instance_id,
- const std::vector<ExprContext*>& output_expr_ctxs,
- RuntimeProfile* parent_profile, BufferControlBlock* sinker,
- Block* output_block, bool output_object_data,
- const RowDescriptor& output_row_descriptor)
+VFileResultWriter::VFileResultWriter(
+ const ResultFileOptions* file_opts, const TStorageBackendType::type storage_type,
+ const TUniqueId fragment_instance_id,
+ const std::vector<vectorized::VExprContext*>& output_vexpr_ctxs,
+ RuntimeProfile* parent_profile, BufferControlBlock* sinker, Block* output_block,
+ bool output_object_data, const RowDescriptor& output_row_descriptor)
: _file_opts(file_opts),
_storage_type(storage_type),
_fragment_instance_id(fragment_instance_id),
- _output_expr_ctxs(output_expr_ctxs),
+ _output_vexpr_ctxs(output_vexpr_ctxs),
_parent_profile(parent_profile),
_sinker(sinker),
_output_block(output_block),
@@ -196,7 +198,16 @@ Status VFileResultWriter::append_block(Block& block) {
if (_parquet_writer != nullptr) {
return Status::NotSupported("Parquet Writer is not supported yet!");
} else {
- RETURN_IF_ERROR(_write_csv_file(block));
+ Status status = Status::OK();
+ // Exec vectorized expr here to speed up, block.rows() == 0 means expr exec
+ // failed, just return the error status
+ auto output_block = VExprContext::get_output_block_after_execute_exprs(_output_vexpr_ctxs,
+ block, status);
+ auto num_rows = output_block.rows();
+ if (UNLIKELY(num_rows == 0)) {
+ return status;
+ }
+ RETURN_IF_ERROR(_write_csv_file(output_block));
}
_written_rows += block.rows();
@@ -210,7 +221,7 @@ Status VFileResultWriter::_write_csv_file(const Block& block) {
if (col.column->is_null_at(i)) {
_plain_text_outstream << NULL_IN_CSV;
} else {
- switch (_output_expr_ctxs[col_id]->root()->type().type) {
+ switch (_output_vexpr_ctxs[col_id]->root()->type().type) {
case TYPE_BOOLEAN:
case TYPE_TINYINT:
_plain_text_outstream << (int)*reinterpret_cast<const int8_t*>(
@@ -280,8 +291,7 @@ Status VFileResultWriter::_write_csv_file(const Block& block) {
reinterpret_cast<const PackedInt128*>(col.column->get_data_at(i).data)
->value);
std::string decimal_str;
- int output_scale = _output_expr_ctxs[col_id]->root()->output_scale();
- decimal_str = decimal_val.to_string(output_scale);
+ decimal_str = decimal_val.to_string();
_plain_text_outstream << decimal_str;
break;
}
diff --git a/be/src/vec/runtime/vfile_result_writer.h b/be/src/vec/runtime/vfile_result_writer.h
index b7fd2cd737..5f0bb7971e 100644
--- a/be/src/vec/runtime/vfile_result_writer.h
+++ b/be/src/vec/runtime/vfile_result_writer.h
@@ -30,22 +30,22 @@ public:
VFileResultWriter(const ResultFileOptions* file_option,
const TStorageBackendType::type storage_type,
const TUniqueId fragment_instance_id,
- const std::vector<ExprContext*>& output_expr_ctxs,
+ const std::vector<VExprContext*>& _output_vexpr_ctxs,
RuntimeProfile* parent_profile, BufferControlBlock* sinker,
Block* output_block, bool output_object_data,
const RowDescriptor& output_row_descriptor);
virtual ~VFileResultWriter() = default;
- virtual Status append_block(Block& block) override;
- virtual Status append_row_batch(const RowBatch* batch) override {
+ Status append_block(Block& block) override;
+ Status append_row_batch(const RowBatch* batch) override {
return Status::NotSupported("append_row_batch is not supported in VFileResultWriter!");
};
- virtual Status init(RuntimeState* state) override;
- virtual Status close() override;
+ Status init(RuntimeState* state) override;
+ Status close() override;
// file result writer always return statistic result in one row
- virtual int64_t get_written_rows() const override { return 1; }
+ int64_t get_written_rows() const override { return 1; }
private:
Status _write_csv_file(const Block& block);
@@ -77,7 +77,7 @@ private:
const ResultFileOptions* _file_opts;
TStorageBackendType::type _storage_type;
TUniqueId _fragment_instance_id;
- const std::vector<ExprContext*>& _output_expr_ctxs;
+ const std::vector<VExprContext*>& _output_vexpr_ctxs;
// If the result file format is plain text, like CSV, this _file_writer is owned by this FileResultWriter.
// If the result file format is Parquet, this _file_writer is owned by _parquet_writer.
diff --git a/be/src/vec/sink/vresult_file_sink.cpp b/be/src/vec/sink/vresult_file_sink.cpp
index 6d8d994585..b939332e39 100644
--- a/be/src/vec/sink/vresult_file_sink.cpp
+++ b/be/src/vec/sink/vresult_file_sink.cpp
@@ -77,9 +77,10 @@ Status VResultFileSink::init(const TDataSink& tsink) {
Status VResultFileSink::prepare_exprs(RuntimeState* state) {
// From the thrift expressions create the real exprs.
- RETURN_IF_ERROR(Expr::create_expr_trees(state->obj_pool(), _t_output_expr, &_output_expr_ctxs));
+ RETURN_IF_ERROR(
+ VExpr::create_expr_trees(state->obj_pool(), _t_output_expr, &_output_vexpr_ctxs));
// Prepare the exprs to run.
- RETURN_IF_ERROR(Expr::prepare(_output_expr_ctxs, state, _row_desc, _expr_mem_tracker));
+ RETURN_IF_ERROR(VExpr::prepare(_output_vexpr_ctxs, state, _row_desc, _expr_mem_tracker));
return Status::OK();
}
@@ -100,7 +101,7 @@ Status VResultFileSink::prepare(RuntimeState* state) {
state->fragment_instance_id(), _buf_size, &_sender));
// create writer
_writer.reset(new (std::nothrow) VFileResultWriter(
- _file_opts.get(), _storage_type, state->fragment_instance_id(), _output_expr_ctxs,
+ _file_opts.get(), _storage_type, state->fragment_instance_id(), _output_vexpr_ctxs,
_profile, _sender.get(), nullptr, state->return_object_data_as_binary(),
_output_row_descriptor));
} else {
@@ -115,7 +116,7 @@ Status VResultFileSink::prepare(RuntimeState* state) {
// create writer
_output_block.reset(new Block(_output_row_descriptor.tuple_descriptors()[0]->slots(), 1));
_writer.reset(new (std::nothrow) VFileResultWriter(
- _file_opts.get(), _storage_type, state->fragment_instance_id(), _output_expr_ctxs,
+ _file_opts.get(), _storage_type, state->fragment_instance_id(), _output_vexpr_ctxs,
_profile, nullptr, _output_block.get(), state->return_object_data_as_binary(),
_output_row_descriptor));
}
@@ -127,7 +128,7 @@ Status VResultFileSink::prepare(RuntimeState* state) {
}
Status VResultFileSink::open(RuntimeState* state) {
- return Expr::open(_output_expr_ctxs, state);
+ return VExpr::open(_output_vexpr_ctxs, state);
}
Status VResultFileSink::send(RuntimeState* state, RowBatch* batch) {
@@ -186,7 +187,7 @@ Status VResultFileSink::close(RuntimeState* state, Status exec_status) {
_output_block->clear();
}
- Expr::close(_output_expr_ctxs, state);
+ VExpr::close(_output_vexpr_ctxs, state);
_closed = true;
return Status::OK();
diff --git a/be/src/vec/sink/vresult_file_sink.h b/be/src/vec/sink/vresult_file_sink.h
index e924883b42..f550fa3585 100644
--- a/be/src/vec/sink/vresult_file_sink.h
+++ b/be/src/vec/sink/vresult_file_sink.h
@@ -34,18 +34,18 @@ public:
const std::vector<TPlanFragmentDestination>& destinations,
int per_channel_buffer_size, bool send_query_statistics_with_every_batch,
const std::vector<TExpr>& t_output_expr, DescriptorTbl& descs);
- virtual ~VResultFileSink() = default;
- virtual Status init(const TDataSink& thrift_sink) override;
- virtual Status prepare(RuntimeState* state) override;
- virtual Status open(RuntimeState* state) override;
+ ~VResultFileSink() override = default;
+ Status init(const TDataSink& thrift_sink) override;
+ Status prepare(RuntimeState* state) override;
+ Status open(RuntimeState* state) override;
// send data in 'batch' to this backend stream mgr
// Blocks until all rows in batch are placed in the buffer
- virtual Status send(RuntimeState* state, RowBatch* batch) override;
- virtual Status send(RuntimeState* state, Block* block) override;
+ Status send(RuntimeState* state, RowBatch* batch) override;
+ Status send(RuntimeState* state, Block* block) override;
// Flush all buffered data and close all existing channels to destination
// hosts. Further send() calls are illegal after calling close().
- virtual Status close(RuntimeState* state, Status exec_status) override;
- virtual RuntimeProfile* profile() override { return _profile; }
+ Status close(RuntimeState* state, Status exec_status) override;
+ RuntimeProfile* profile() override { return _profile; }
void set_query_statistics(std::shared_ptr<QueryStatistics> statistics) override;
@@ -57,7 +57,7 @@ private:
// Owned by the RuntimeState.
const std::vector<TExpr>& _t_output_expr;
- std::vector<ExprContext*> _output_expr_ctxs;
+ std::vector<vectorized::VExprContext*> _output_vexpr_ctxs;
RowDescriptor _output_row_descriptor;
std::unique_ptr<Block> _output_block = nullptr;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org