You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2023/07/08 14:25:18 UTC
[doris] branch branch-1.2-lts updated (5d1301afff -> 3eca5cb61c)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
from 5d1301afff [typo](docs) modify java version to 1.8 (#21538)
add 9fd334e39e print more info
add 4044eea3aa fix bug of tebu
add 71a299587d Merge pull request #2 from starocean999/yiguolei_1.2
add f3671946c1 fix tebu bug 2
new 68369ff776 Merge pull request #3 from starocean999/yiguolei_1.2
new 5b3eeb49f0 add exec node id and column info
new bb9925e0e2 f
new 3eca5cb61c [Fix](storage)read page cache when seek #21272
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 1 +
be/src/vec/columns/column_string.cpp | 10 +++-
be/src/vec/common/assert_cast.h | 17 ++++++
be/src/vec/exec/join/vjoin_node_base.cpp | 7 +++
.../apache/doris/analysis/ExprSubstitutionMap.java | 35 +++++++++---
.../org/apache/doris/planner/JoinNodeBase.java | 64 ++++++++++++++--------
6 files changed, 99 insertions(+), 35 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 04/04: [Fix](storage)read page cache when seek #21272
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3eca5cb61ca70dd0c7c3e87ce1f004fd720b7167
Author: wangbo <wa...@apache.org>
AuthorDate: Wed Jun 28 15:53:40 2023 +0800
[Fix](storage)read page cache when seek #21272
Currently, when a columnIter is used for seek, then page cache is not set;
When this colunIter is used for later read data, then page cache could not be used.
---
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 1 +
1 file changed, 1 insertion(+)
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index 94fc3dfa5d..cb9dff43c1 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -289,6 +289,7 @@ Status SegmentIterator::_prepare_seek(const StorageReadOptions::KeyRange& key_ra
&_column_iterators[unique_id]));
ColumnIteratorOptions iter_opts;
iter_opts.stats = _opts.stats;
+ iter_opts.use_page_cache = _opts.use_page_cache;
iter_opts.file_reader = _file_reader.get();
iter_opts.io_ctx = _opts.io_ctx;
RETURN_IF_ERROR(_column_iterators[unique_id]->init(iter_opts));
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 01/04: Merge pull request #3 from starocean999/yiguolei_1.2
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 68369ff776832cae784cc934c55a449b246f351c
Merge: 71a299587d f3671946c1
Author: yiguolei <67...@qq.com>
AuthorDate: Fri Jul 7 16:05:15 2023 +0800
Merge pull request #3 from starocean999/yiguolei_1.2
fix bug of tebu
.../main/java/org/apache/doris/analysis/ExprSubstitutionMap.java | 9 +++++++++
.../src/main/java/org/apache/doris/planner/JoinNodeBase.java | 1 +
2 files changed, 10 insertions(+)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 03/04: f
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit bb9925e0e2ecf833a0674a054c3c9cf644613957
Author: yiguolei <yi...@gmail.com>
AuthorDate: Fri Jul 7 17:16:51 2023 +0800
f
---
.../apache/doris/analysis/ExprSubstitutionMap.java | 28 +++++++----
.../org/apache/doris/planner/JoinNodeBase.java | 58 ++++++++++++----------
2 files changed, 51 insertions(+), 35 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java
index 65f9d2f8e0..b8008fde77 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java
@@ -33,11 +33,15 @@ import java.util.Objects;
/**
* Map of expression substitutions: lhs[i] gets substituted with rhs[i].
- * To support expression substitution across query blocks, rhs exprs must already be
- * analyzed when added to this map. Otherwise, analysis of a SlotRef may fail after
- * substitution, e.g., because the table it refers to is in a different query block
+ * To support expression substitution across query blocks, rhs exprs must
+ * already be
+ * analyzed when added to this map. Otherwise, analysis of a SlotRef may fail
+ * after
+ * substitution, e.g., because the table it refers to is in a different query
+ * block
* that is not visible.
- * See Expr.substitute() and related functions for details on the actual substitution.
+ * See Expr.substitute() and related functions for details on the actual
+ * substitution.
*/
public final class ExprSubstitutionMap {
private static final Logger LOG = LogManager.getLogger(ExprSubstitutionMap.class);
@@ -62,7 +66,8 @@ public final class ExprSubstitutionMap {
}
/**
- * Add an expr mapping. The rhsExpr must be analyzed to support correct substitution
+ * Add an expr mapping. The rhsExpr must be analyzed to support correct
+ * substitution
* across query blocks. It is not required that the lhsExpr is analyzed.
*/
public void put(Expr lhsExpr, Expr rhsExpr) {
@@ -132,7 +137,7 @@ public final class ExprSubstitutionMap {
}
/**
- * Return a map which is equivalent to applying f followed by g,
+ * Return a map which is equivalent to applying f followed by g,
* i.e., g(f()).
* Always returns a non-null map.
*/
@@ -208,7 +213,8 @@ public final class ExprSubstitutionMap {
/**
* Returns the replace of two substitution maps.
- * f [A.id, B.id] [A.age, B.age] [A.name, B.name] g [A.id, C.id] [B.age, C.age] [A.address, C.address]
+ * f [A.id, B.id] [A.age, B.age] [A.name, B.name] g [A.id, C.id] [B.age, C.age]
+ * [A.address, C.address]
* return: [A.id, C,id] [A.age, C.age] [A.name, B.name] [A.address, C.address]
*/
public static ExprSubstitutionMap composeAndReplace(ExprSubstitutionMap f, ExprSubstitutionMap g, Analyzer analyzer)
@@ -304,11 +310,13 @@ public final class ExprSubstitutionMap {
}
/**
- * Verifies the internal state of this smap: Checks that the lhs_ has no duplicates,
+ * Verifies the internal state of this smap: Checks that the lhs_ has no
+ * duplicates,
* and that all rhs exprs are analyzed.
*/
private void verify() {
- // This method is very very time consuming, especially when planning large complex query.
+ // This method is very very time consuming, especially when planning large
+ // complex query.
// So disable it by default.
if (LOG.isDebugEnabled()) {
for (int i = 0; i < lhs.size(); ++i) {
@@ -340,7 +348,7 @@ public final class ExprSubstitutionMap {
Preconditions.checkState(lhs.size() == rhs.size(), "lhs and rhs must be same size");
for (int i = 0; i < rhs.size(); i++) {
if (rhs.get(i) instanceof SlotRef) {
- ((SlotRef) rhs.get(i)).getDesc().setIsNullable(lhs.get(i).isNullable());
+ ((SlotRef) rhs.get(i)).getDesc().setIsNullable(lhs.get(i).isNullable() || ((SlotRef) rhs.get(i)).getDesc().getIsNullable());
}
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java b/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java
index 19921125e3..bdf5c0f161 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java
@@ -158,21 +158,22 @@ public abstract class JoinNodeBase extends PlanNode {
int leftNullableNumber = 0;
int rightNullableNumber = 0;
if (copyLeft) {
- //cross join do not have OutputTblRefIds
+ // cross join do not have OutputTblRefIds
List<TupleId> srcTupleIds = getChild(0) instanceof JoinNodeBase ? getChild(0).getOutputTupleIds()
: getChild(0).getOutputTblRefIds();
for (TupleDescriptor leftTupleDesc : analyzer.getDescTbl().getTupleDesc(srcTupleIds)) {
- // if the child is cross join node, the only way to get the correct nullable info of its output slots
+ // if the child is cross join node, the only way to get the correct nullable
+ // info of its output slots
// is to check if the output tuple ids are outer joined or not.
// then pass this nullable info to hash join node will be correct.
- boolean needSetToNullable =
- getChild(0) instanceof JoinNodeBase && analyzer.isOuterJoined(leftTupleDesc.getId());
+ boolean needSetToNullable = getChild(0) instanceof JoinNodeBase
+ && analyzer.isOuterJoined(leftTupleDesc.getId());
for (SlotDescriptor leftSlotDesc : leftTupleDesc.getSlots()) {
if (!isMaterializedByChild(leftSlotDesc, getChild(0).getOutputSmap())) {
continue;
}
- SlotDescriptor outputSlotDesc =
- analyzer.getDescTbl().copySlotDescriptor(vOutputTupleDesc, leftSlotDesc);
+ SlotDescriptor outputSlotDesc = analyzer.getDescTbl().copySlotDescriptor(vOutputTupleDesc,
+ leftSlotDesc);
if (leftNullable) {
outputSlotDesc.setIsNullable(true);
leftNullableNumber++;
@@ -188,14 +189,14 @@ public abstract class JoinNodeBase extends PlanNode {
List<TupleId> srcTupleIds = getChild(1) instanceof JoinNodeBase ? getChild(1).getOutputTupleIds()
: getChild(1).getOutputTblRefIds();
for (TupleDescriptor rightTupleDesc : analyzer.getDescTbl().getTupleDesc(srcTupleIds)) {
- boolean needSetToNullable =
- getChild(1) instanceof JoinNodeBase && analyzer.isOuterJoined(rightTupleDesc.getId());
+ boolean needSetToNullable = getChild(1) instanceof JoinNodeBase
+ && analyzer.isOuterJoined(rightTupleDesc.getId());
for (SlotDescriptor rightSlotDesc : rightTupleDesc.getSlots()) {
if (!isMaterializedByChild(rightSlotDesc, getChild(1).getOutputSmap())) {
continue;
}
- SlotDescriptor outputSlotDesc =
- analyzer.getDescTbl().copySlotDescriptor(vOutputTupleDesc, rightSlotDesc);
+ SlotDescriptor outputSlotDesc = analyzer.getDescTbl().copySlotDescriptor(vOutputTupleDesc,
+ rightSlotDesc);
if (rightNullable) {
outputSlotDesc.setIsNullable(true);
rightNullableNumber++;
@@ -211,8 +212,7 @@ public abstract class JoinNodeBase extends PlanNode {
// add mark slot if needed
if (isMarkJoin() && analyzer.needPopUpMarkTuple(innerRef)) {
SlotDescriptor markSlot = analyzer.getMarkTuple(innerRef).getSlots().get(0);
- SlotDescriptor outputSlotDesc =
- analyzer.getDescTbl().copySlotDescriptor(vOutputTupleDesc, markSlot);
+ SlotDescriptor outputSlotDesc = analyzer.getDescTbl().copySlotDescriptor(vOutputTupleDesc, markSlot);
srcTblRefToOutputTupleSmap.put(new SlotRef(markSlot), new SlotRef(outputSlotDesc));
}
@@ -313,8 +313,10 @@ public abstract class JoinNodeBase extends PlanNode {
}
}
- // be may be possible to output correct row number without any column data in future
- // but for now, in order to have correct output row number, should keep at least one slot.
+ // be may be possible to output correct row number without any column data in
+ // future
+ // but for now, in order to have correct output row number, should keep at least
+ // one slot.
// use first materialized slot if outputSlotIds is empty.
if (outputSlotIds.isEmpty() && firstMaterializedSlotId != null) {
outputSlotIds.add(firstMaterializedSlotId);
@@ -359,7 +361,8 @@ public abstract class JoinNodeBase extends PlanNode {
vIntermediateTupleDescList = new ArrayList<>();
vIntermediateTupleDescList.add(vIntermediateLeftTupleDesc);
vIntermediateTupleDescList.add(vIntermediateRightTupleDesc);
- // if join type is MARK, add mark tuple to intermediate tuple. mark slot will be generated after join.
+ // if join type is MARK, add mark tuple to intermediate tuple. mark slot will be
+ // generated after join.
if (isMarkJoin()) {
TupleDescriptor markTuple = analyzer.getMarkTuple(innerRef);
if (markTuple != null) {
@@ -395,8 +398,8 @@ public abstract class JoinNodeBase extends PlanNode {
for (TupleDescriptor tupleDescriptor : analyzer.getDescTbl()
.getTupleDesc(getChild(0).getOutputTupleIds())) {
for (SlotDescriptor slotDescriptor : tupleDescriptor.getMaterializedSlots()) {
- SlotDescriptor intermediateSlotDesc =
- analyzer.getDescTbl().copySlotDescriptor(vIntermediateLeftTupleDesc, slotDescriptor);
+ SlotDescriptor intermediateSlotDesc = analyzer.getDescTbl()
+ .copySlotDescriptor(vIntermediateLeftTupleDesc, slotDescriptor);
if (leftNullable) {
intermediateSlotDesc.setIsNullable(true);
}
@@ -411,8 +414,8 @@ public abstract class JoinNodeBase extends PlanNode {
for (TupleDescriptor tupleDescriptor : analyzer.getDescTbl()
.getTupleDesc(getChild(1).getOutputTupleIds())) {
for (SlotDescriptor slotDescriptor : tupleDescriptor.getMaterializedSlots()) {
- SlotDescriptor intermediateSlotDesc =
- analyzer.getDescTbl().copySlotDescriptor(vIntermediateRightTupleDesc, slotDescriptor);
+ SlotDescriptor intermediateSlotDesc = analyzer.getDescTbl()
+ .copySlotDescriptor(vIntermediateRightTupleDesc, slotDescriptor);
if (rightNullable) {
intermediateSlotDesc.setIsNullable(true);
}
@@ -423,7 +426,8 @@ public abstract class JoinNodeBase extends PlanNode {
vIntermediateRightTupleDesc.computeMemLayout();
// 3. replace srcExpr by intermediate tuple
Preconditions.checkState(vSrcToOutputSMap != null);
- // Set `preserveRootTypes` to true because we should keep the consistent for types. See Issue-11314.
+ // Set `preserveRootTypes` to true because we should keep the consistent for
+ // types. See Issue-11314.
vSrcToOutputSMap.substituteLhs(originToIntermediateSmap, analyzer, true);
// 4. replace other conjuncts and conjuncts
computeOtherConjuncts(analyzer, originToIntermediateSmap);
@@ -438,7 +442,7 @@ public abstract class JoinNodeBase extends PlanNode {
List<Expr> exprs = vSrcToOutputSMap.getLhs();
ArrayList<SlotDescriptor> slots = vOutputTupleDesc.getSlots();
for (int i = 0; i < slots.size(); i++) {
- slots.get(i).setIsNullable(exprs.get(i).isNullable());
+ slots.get(i).setIsNullable(exprs.get(i).isNullable() || slots.get(i).getIsNullable());
}
vSrcToOutputSMap.reCalculateNullableInfoForSlotInRhs();
}
@@ -509,7 +513,8 @@ public abstract class JoinNodeBase extends PlanNode {
/**
* If parent wants to get join node tupleids,
* it will call this function instead of read properties directly.
- * The reason is that the tuple id of vOutputTupleDesc the real output tuple id for join node.
+ * The reason is that the tuple id of vOutputTupleDesc the real output tuple id
+ * for join node.
* <p>
* If you read the properties of @tupleids directly instead of this function,
* it reads the input id of the current node.
@@ -606,12 +611,15 @@ public abstract class JoinNodeBase extends PlanNode {
newRhs.add(rhsExpr);
} else {
// we need do project in the join node
- // add a new slot for projection result and add the project expr to vSrcToOutputSMap
+ // add a new slot for projection result and add the project expr to
+ // vSrcToOutputSMap
SlotDescriptor slotDesc = analyzer.addSlotDescriptor(vOutputTupleDesc);
slotDesc.initFromExpr(rhsExpr);
slotDesc.setIsMaterialized(true);
- // the project expr is from smap, which use the slots of hash join node's output tuple
- // we need substitute it to make sure the project expr use slots from intermediate tuple
+ // the project expr is from smap, which use the slots of hash join node's output
+ // tuple
+ // we need substitute it to make sure the project expr use slots from
+ // intermediate tuple
rhsExpr = rhsExpr.substitute(tmpSmap);
vSrcToOutputSMap.getLhs().add(rhsExpr);
SlotRef slotRef = new SlotRef(slotDesc);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 02/04: add exec node id and column info
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5b3eeb49f0a25c6669eb8396b6e57cf5af63b134
Author: yiguolei <yi...@gmail.com>
AuthorDate: Fri Jul 7 17:10:13 2023 +0800
add exec node id and column info
---
be/src/vec/exec/join/vjoin_node_base.cpp | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/be/src/vec/exec/join/vjoin_node_base.cpp b/be/src/vec/exec/join/vjoin_node_base.cpp
index 869a50381e..8a2232691f 100644
--- a/be/src/vec/exec/join/vjoin_node_base.cpp
+++ b/be/src/vec/exec/join/vjoin_node_base.cpp
@@ -140,6 +140,11 @@ Status VJoinNodeBase::_build_output_block(Block* origin_block, Block* output_blo
RETURN_IF_ERROR(_output_expr_ctxs[i]->execute(origin_block, &result_column_id));
auto column_ptr = origin_block->get_by_position(result_column_id)
.column->convert_to_full_column_if_const();
+ // check nullable here, and print more info about plan
+ if (!mutable_columns[i]->is_nullable() && column_ptr->is_nullable()) {
+ LOG(FATAL) << "to is not nullable while from is nullable, could not copy. i = "
+ << i << ", exec_node_id = " << _id;
+ }
insert_column_datas(mutable_columns[i], *column_ptr, rows);
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org