You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2024/02/22 05:02:14 UTC
(doris) branch branch-2.1 updated (6bbe6c9a591 -> ef22bd3318c)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
from 6bbe6c9a591 [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. #31212
new 57f3ea6d7bf Example Modify audit log documents (#31192)
new e183e9ac460 [fix](stream-load) print stream load profile for pipeline and pipelinex (#31198)
new 2069f9ff63d [fix](Nereids): OuterJoinAssoc reject literal condition (#31172)
new 9c096710fa3 [Enhancement](group commit) Add bvar and log for group commit (#31017)
new 8a496bf554d [regression test]Optimize testUniqueModelSchemaKeyChange (#31150)
new c56cb0ac3e5 [Exec](RF) Support merge remote rf local first (#31067)
new 98a1b12ac6d [Enhancement] [plsql] support drop store procedure (#30966) (#31001)
new f2a38e63456 [chore](columns) remove update_hashes_with_value for SipHash (#31224)
new 70dc40ab3c8 [fix](audit) The time field in the audit log table is set to the millisecond level (#31096)
new 6ef90b2ad96 Window function sum/avg does not support distinct document modification (#31066)
new d080fe8c89a [log](query) add debug log for check query visible version (#31217)
new b66583551c3 [fix](group_commit)Fix bound checking problem when reading wal block (#31112)
new bb73561b712 [doc](create table) add since version for some new features (#30506)
new 52b9af06fb4 [pipelineX](refactor) Delete subclasses inherited from Dependency (#31216)
new 90a7f04349a [refine](pipelinex) get sink local state does not require an id. #31195
new 6096227fea3 [bugfix](docs)Fix some spelling mistakes in english doc (#31241)
new ad07dec0ed5 [Improve](InPredict) enhance in predict with struct type (#30840)
new 90ab5ec2d99 [fix](invert index) fix the error issue in the unit test remove_element_only_in_table (#31238)
new ef22bd3318c [fix](txn insert) Txn insert can not write to table with mv (#31167)
The 19 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/agent/be_exec_version_manager.h | 1 +
be/src/exprs/runtime_filter.cpp | 77 +++---
be/src/exprs/runtime_filter.h | 24 +-
be/src/exprs/runtime_filter_slots.h | 11 +-
be/src/pipeline/exec/aggregation_sink_operator.h | 12 +-
be/src/pipeline/exec/aggregation_source_operator.h | 12 +-
be/src/pipeline/exec/analytic_sink_operator.cpp | 2 +-
be/src/pipeline/exec/analytic_sink_operator.h | 12 +-
be/src/pipeline/exec/analytic_source_operator.cpp | 6 +-
be/src/pipeline/exec/analytic_source_operator.h | 10 +-
be/src/pipeline/exec/assert_num_rows_operator.h | 4 +-
be/src/pipeline/exec/data_queue.h | 5 -
be/src/pipeline/exec/datagen_operator.cpp | 9 +-
.../distinct_streaming_aggregation_operator.cpp | 2 +-
.../exec/distinct_streaming_aggregation_operator.h | 4 +-
be/src/pipeline/exec/empty_set_operator.h | 4 +-
be/src/pipeline/exec/es_scan_operator.cpp | 2 +-
be/src/pipeline/exec/exchange_sink_buffer.h | 5 +-
be/src/pipeline/exec/exchange_sink_operator.cpp | 5 +-
be/src/pipeline/exec/exchange_sink_operator.h | 62 ++---
be/src/pipeline/exec/exchange_source_operator.cpp | 4 +-
be/src/pipeline/exec/exchange_source_operator.h | 26 +-
be/src/pipeline/exec/file_scan_operator.cpp | 2 +-
be/src/pipeline/exec/hashjoin_build_sink.cpp | 28 +--
be/src/pipeline/exec/hashjoin_build_sink.h | 18 +-
be/src/pipeline/exec/hashjoin_probe_operator.cpp | 4 +-
be/src/pipeline/exec/hashjoin_probe_operator.h | 10 +-
be/src/pipeline/exec/join_build_sink_operator.cpp | 20 +-
be/src/pipeline/exec/join_build_sink_operator.h | 6 +-
be/src/pipeline/exec/join_probe_operator.cpp | 24 +-
be/src/pipeline/exec/join_probe_operator.h | 8 +-
be/src/pipeline/exec/multi_cast_data_stream_sink.h | 12 +-
.../exec/multi_cast_data_stream_source.cpp | 3 +-
.../pipeline/exec/multi_cast_data_stream_source.h | 15 +-
be/src/pipeline/exec/multi_cast_data_streamer.h | 6 +-
.../exec/nested_loop_join_build_operator.cpp | 8 +-
.../exec/nested_loop_join_build_operator.h | 10 +-
.../exec/nested_loop_join_probe_operator.cpp | 6 +-
.../exec/nested_loop_join_probe_operator.h | 10 +-
be/src/pipeline/exec/olap_scan_operator.cpp | 4 +-
.../pipeline/exec/partition_sort_sink_operator.cpp | 2 +-
.../pipeline/exec/partition_sort_sink_operator.h | 12 +-
.../exec/partition_sort_source_operator.cpp | 2 +-
.../pipeline/exec/partition_sort_source_operator.h | 14 +-
be/src/pipeline/exec/repeat_operator.h | 4 +-
be/src/pipeline/exec/result_sink_operator.cpp | 3 +-
be/src/pipeline/exec/result_sink_operator.h | 13 +-
be/src/pipeline/exec/scan_operator.cpp | 24 +-
be/src/pipeline/exec/scan_operator.h | 47 +---
be/src/pipeline/exec/select_operator.h | 4 +-
be/src/pipeline/exec/set_probe_sink_operator.cpp | 5 +-
be/src/pipeline/exec/set_probe_sink_operator.h | 21 +-
be/src/pipeline/exec/set_sink_operator.cpp | 4 +-
be/src/pipeline/exec/set_sink_operator.h | 16 +-
be/src/pipeline/exec/set_source_operator.cpp | 2 +-
be/src/pipeline/exec/set_source_operator.h | 12 +-
be/src/pipeline/exec/sort_sink_operator.cpp | 2 +-
be/src/pipeline/exec/sort_sink_operator.h | 12 +-
be/src/pipeline/exec/sort_source_operator.cpp | 2 +-
be/src/pipeline/exec/sort_source_operator.h | 10 +-
.../pipeline/exec/streaming_aggregation_operator.h | 4 +-
be/src/pipeline/exec/union_sink_operator.h | 12 +-
be/src/pipeline/exec/union_source_operator.h | 12 +-
be/src/pipeline/pipeline_fragment_context.cpp | 20 +-
be/src/pipeline/pipeline_fragment_context.h | 2 +-
be/src/pipeline/pipeline_x/dependency.cpp | 2 +-
be/src/pipeline/pipeline_x/dependency.h | 46 +++-
.../local_exchange/local_exchange_sink_operator.h | 13 +-
.../local_exchange_source_operator.cpp | 11 -
.../local_exchange_source_operator.h | 15 +-
.../pipeline_x/local_exchange/local_exchanger.h | 1 -
be/src/pipeline/pipeline_x/operator.cpp | 168 ++++++-------
be/src/pipeline/pipeline_x/operator.h | 46 ++--
.../pipeline_x/pipeline_x_fragment_context.cpp | 38 ++-
.../pipeline_x/pipeline_x_fragment_context.h | 7 +-
be/src/pipeline/pipeline_x/pipeline_x_task.cpp | 22 +-
be/src/pipeline/pipeline_x/pipeline_x_task.h | 7 +-
be/src/runtime/buffer_control_block.cpp | 2 +-
be/src/runtime/buffer_control_block.h | 6 +-
be/src/runtime/fragment_mgr.cpp | 5 +-
be/src/runtime/group_commit_mgr.cpp | 4 +
be/src/runtime/group_commit_mgr.h | 4 +
be/src/runtime/query_context.h | 3 +
be/src/runtime/runtime_filter_mgr.cpp | 91 +++++--
be/src/runtime/runtime_filter_mgr.h | 32 ++-
be/src/runtime/runtime_state.cpp | 42 +++-
be/src/runtime/runtime_state.h | 26 +-
be/src/vec/columns/column.h | 22 --
be/src/vec/columns/column_array.cpp | 5 -
be/src/vec/columns/column_array.h | 3 -
be/src/vec/columns/column_complex.h | 11 +-
be/src/vec/columns/column_const.cpp | 16 --
be/src/vec/columns/column_const.h | 3 -
be/src/vec/columns/column_decimal.cpp | 6 -
be/src/vec/columns/column_decimal.h | 2 -
be/src/vec/columns/column_map.cpp | 5 -
be/src/vec/columns/column_map.h | 3 -
be/src/vec/columns/column_nullable.cpp | 19 --
be/src/vec/columns/column_nullable.h | 2 -
be/src/vec/columns/column_string.h | 5 -
be/src/vec/columns/column_struct.cpp | 5 -
be/src/vec/columns/column_struct.h | 3 -
be/src/vec/columns/column_vector.cpp | 6 -
be/src/vec/columns/column_vector.h | 3 -
be/src/vec/exec/format/wal/wal_reader.cpp | 21 +-
be/src/vec/exec/join/vhash_join_node.cpp | 6 +-
be/src/vec/exec/join/vnested_loop_join_node.cpp | 4 +-
be/src/vec/exec/runtime_filter_consumer.cpp | 23 +-
be/src/vec/exec/runtime_filter_consumer.h | 4 +-
be/src/vec/exec/scan/pip_scanner_context.h | 6 +-
be/src/vec/exec/scan/scanner_context.h | 1 -
be/src/vec/exec/vdata_gen_scan_node.cpp | 9 +-
be/src/vec/exprs/vin_predicate.cpp | 3 +
.../{function_cast.cpp => function_struct_in.cpp} | 11 +-
be/src/vec/functions/function_struct_in.h | 169 +++++++++++++
be/src/vec/functions/in.h | 3 +-
be/src/vec/functions/simple_function_factory.h | 2 +
be/src/vec/runtime/shared_hash_table_controller.h | 7 +-
be/src/vec/runtime/vdata_stream_recvr.cpp | 10 +-
be/src/vec/runtime/vdata_stream_recvr.h | 18 +-
be/src/vec/sink/group_commit_block_sink.cpp | 8 +-
be/src/vec/sink/vdata_stream_sender.cpp | 3 +-
be/src/vec/sink/vdata_stream_sender.h | 4 +-
.../inverted_index_searcher_cache_test.cpp | 2 +-
be/test/vec/columns/column_hash_func_test.cpp | 59 ++---
be/test/vec/exec/vwal_scanner_test.cpp | 155 +++++++-----
.../community/design/doris_storage_optimization.md | 4 +-
.../developer-guide/cpp-diagnostic-code.md | 2 +-
.../commit-format-specification.md | 2 +-
docs/en/community/join-community.md | 2 +-
.../release-and-verify/release-complete.md | 2 +-
.../http-actions/fe/debug-point-action.md | 6 +-
.../admin-manual/http-actions/fe/metrics-action.md | 2 +-
docs/en/docs/admin-manual/privilege-ldap/ranger.md | 2 +-
docs/en/docs/advanced/compute-node.md | 2 +-
docs/en/docs/advanced/partition/auto-partition.md | 2 +-
docs/en/docs/ecosystem/audit-plugin.md | 2 +-
.../window-functions/window-function-avg.md | 2 +-
.../window-functions/window-function-sum.md | 2 +-
.../Create/CREATE-TABLE.md | 4 +-
docs/zh-CN/docs/ecosystem/audit-plugin.md | 2 +-
.../window-functions/window-function-avg.md | 2 +-
.../window-functions/window-function-sum.md | 2 +-
.../Create/CREATE-TABLE.md | 4 +-
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 1 +
.../antlr4/org/apache/doris/nereids/PLParser.g4 | 4 +
.../apache/doris/analysis/NativeInsertStmt.java | 54 +++--
.../org/apache/doris/catalog/InternalSchema.java | 2 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 9 +
.../rules/exploration/join/OuterJoinAssoc.java | 2 +-
.../exploration/join/OuterJoinAssocProject.java | 2 +-
.../nereids/trees/expressions/InPredicate.java | 10 +
.../apache/doris/nereids/trees/plans/PlanType.java | 3 +-
...edureCommand.java => DropProcedureCommand.java} | 20 +-
.../trees/plans/visitor/CommandVisitor.java | 5 +
.../doris/nereids/util/TypeCoercionUtils.java | 11 +-
.../org/apache/doris/planner/OlapScanNode.java | 5 +
.../src/main/java/org/apache/doris/plsql/Exec.java | 33 +++
.../doris/plugin/audit/AuditLoaderPlugin.java | 2 +-
.../main/java/org/apache/doris/qe/Coordinator.java | 3 +-
regression-test/data/insert_p0/txn_insert.out | 12 +
.../nereids_syntax_p0/inpredicate_with_struct.out | 19 ++
.../test_unique_model_schema_key_change.out | 121 +++++++++
regression-test/suites/insert_p0/txn_insert.groovy | 21 +-
...pdiff.groovy => inpredicate_with_struct.groovy} | 35 +--
.../suites/plsql_p0/test_plsql_loop_cursor.groovy | 4 +
.../test_unique_model_schema_key_change.groovy | 270 +++++++++++++++++++++
167 files changed, 1562 insertions(+), 1124 deletions(-)
copy be/src/vec/functions/{function_cast.cpp => function_struct_in.cpp} (79%)
create mode 100644 be/src/vec/functions/function_struct_in.h
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/{CreateProcedureCommand.java => DropProcedureCommand.java} (68%)
create mode 100644 regression-test/data/nereids_syntax_p0/inpredicate_with_struct.out
create mode 100644 regression-test/data/schema_change_p0/test_unique_model_schema_key_change.out
copy regression-test/suites/nereids_syntax_p0/{test_timestampdiff.groovy => inpredicate_with_struct.groovy} (52%)
create mode 100644 regression-test/suites/schema_change_p0/test_unique_model_schema_key_change.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 04/19: [Enhancement](group commit) Add bvar and log for group commit (#31017)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 9c096710fa396722270caeb5c322d6bf8a432cc7
Author: abmdocrt <Yu...@gmail.com>
AuthorDate: Wed Feb 21 20:29:25 2024 +0800
[Enhancement](group commit) Add bvar and log for group commit (#31017)
---
be/src/runtime/group_commit_mgr.cpp | 4 ++++
be/src/runtime/group_commit_mgr.h | 4 ++++
be/src/vec/sink/group_commit_block_sink.cpp | 8 ++++++--
3 files changed, 14 insertions(+), 2 deletions(-)
diff --git a/be/src/runtime/group_commit_mgr.cpp b/be/src/runtime/group_commit_mgr.cpp
index 7422890aa41..42c0e83f4b6 100644
--- a/be/src/runtime/group_commit_mgr.cpp
+++ b/be/src/runtime/group_commit_mgr.cpp
@@ -78,6 +78,7 @@ Status LoadBlockQueue::add_block(RuntimeState* runtime_state,
VLOG_DEBUG << "group commit meets commit condition for data size, label=" << label
<< ", instance_id=" << load_instance_id << ", data_bytes=" << _data_bytes;
_need_commit = true;
+ data_size_condition = true;
}
_get_cond.notify_all();
return Status::OK();
@@ -417,6 +418,9 @@ Status GroupCommitTable::_finish_group_commit_load(int64_t db_id, int64_t table_
<< ", exec_plan_fragment status=" << status.to_string()
<< ", commit/abort txn rpc status=" << st.to_string()
<< ", commit/abort txn status=" << result_status.to_string()
+ << ", this group commit includes " << load_block_queue->group_commit_load_count << " loads"
+ << ", flush because meet "
+ << (load_block_queue->data_size_condition ? "data size " : "time ") << "condition"
<< ", wal space info:" << ExecEnv::GetInstance()->wal_mgr()->get_wal_dirs_info_string();
if (state) {
if (!state->get_error_log_file_path().empty()) {
diff --git a/be/src/runtime/group_commit_mgr.h b/be/src/runtime/group_commit_mgr.h
index e3b28be5804..5357ba208f7 100644
--- a/be/src/runtime/group_commit_mgr.h
+++ b/be/src/runtime/group_commit_mgr.h
@@ -78,6 +78,10 @@ public:
int64_t txn_id;
int64_t schema_version;
bool wait_internal_group_commit_finish = false;
+ bool data_size_condition = false;
+
+ // counts of load in one group commit
+ std::atomic_size_t group_commit_load_count = 0;
// the execute status of this internal group commit
std::mutex mutex;
diff --git a/be/src/vec/sink/group_commit_block_sink.cpp b/be/src/vec/sink/group_commit_block_sink.cpp
index 6ff9d4a1425..2ce03ba0c1f 100644
--- a/be/src/vec/sink/group_commit_block_sink.cpp
+++ b/be/src/vec/sink/group_commit_block_sink.cpp
@@ -35,6 +35,8 @@
namespace doris {
namespace vectorized {
+bvar::Adder<int64_t> g_group_commit_load_rows("doris_group_commit_load_rows");
+bvar::Adder<int64_t> g_group_commit_load_bytes("doris_group_commit_load_bytes");
GroupCommitBlockSink::GroupCommitBlockSink(ObjectPool* pool, const RowDescriptor& row_desc,
const std::vector<TExpr>& texprs, Status* status)
@@ -48,6 +50,7 @@ GroupCommitBlockSink::~GroupCommitBlockSink() {
if (_load_block_queue) {
_remove_estimated_wal_bytes();
_load_block_queue->remove_load_id(_load_id);
+ _load_block_queue->group_commit_load_count.fetch_add(1);
}
}
@@ -144,12 +147,13 @@ Status GroupCommitBlockSink::send(RuntimeState* state, vectorized::Block* input_
return status;
}
SCOPED_TIMER(_profile->total_time_counter());
+
// update incrementally so that FE can get the progress.
// the real 'num_rows_load_total' will be set when sink being closed.
state->update_num_rows_load_total(rows);
state->update_num_bytes_load_total(bytes);
- DorisMetrics::instance()->load_rows->increment(rows);
- DorisMetrics::instance()->load_bytes->increment(bytes);
+ g_group_commit_load_rows << rows;
+ g_group_commit_load_bytes << bytes;
std::shared_ptr<vectorized::Block> block;
bool has_filtered_rows = false;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 08/19: [chore](columns) remove update_hashes_with_value for SipHash (#31224)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f2a38e634567c26a981996e84133f6f5aa59c0d3
Author: Uniqueyou <13...@users.noreply.github.com>
AuthorDate: Wed Feb 21 23:56:50 2024 +0800
[chore](columns) remove update_hashes_with_value for SipHash (#31224)
---
be/src/vec/columns/column.h | 22 ------------------
be/src/vec/columns/column_array.cpp | 5 ----
be/src/vec/columns/column_array.h | 3 ---
be/src/vec/columns/column_complex.h | 11 ++-------
be/src/vec/columns/column_const.cpp | 16 -------------
be/src/vec/columns/column_const.h | 3 ---
be/src/vec/columns/column_decimal.cpp | 6 -----
be/src/vec/columns/column_decimal.h | 2 --
be/src/vec/columns/column_map.cpp | 5 ----
be/src/vec/columns/column_map.h | 3 ---
be/src/vec/columns/column_nullable.cpp | 19 ---------------
be/src/vec/columns/column_nullable.h | 2 --
be/src/vec/columns/column_string.h | 5 ----
be/src/vec/columns/column_struct.cpp | 5 ----
be/src/vec/columns/column_struct.h | 3 ---
be/src/vec/columns/column_vector.cpp | 6 -----
be/src/vec/columns/column_vector.h | 3 ---
be/test/vec/columns/column_hash_func_test.cpp | 33 ---------------------------
18 files changed, 2 insertions(+), 150 deletions(-)
diff --git a/be/src/vec/columns/column.h b/be/src/vec/columns/column.h
index 322456a8f77..bf869961544 100644
--- a/be/src/vec/columns/column.h
+++ b/be/src/vec/columns/column.h
@@ -46,19 +46,6 @@
class SipHash;
-#define SIP_HASHES_FUNCTION_COLUMN_IMPL() \
- auto s = hashes.size(); \
- DCHECK(s == size()); \
- if (null_data == nullptr) { \
- for (size_t i = 0; i < s; i++) { \
- update_hash_with_value(i, hashes[i]); \
- } \
- } else { \
- for (size_t i = 0; i < s; i++) { \
- if (null_data[i] == 0) update_hash_with_value(i, hashes[i]); \
- } \
- }
-
#define DO_CRC_HASHES_FUNCTION_COLUMN_IMPL() \
if (null_data == nullptr) { \
for (size_t i = 0; i < s; i++) { \
@@ -362,15 +349,6 @@ public:
LOG(FATAL) << get_name() << " update_hash_with_value siphash not supported";
}
- /// Update state of hash function with value of n elements to avoid the virtual function call
- /// null_data to mark whether need to do hash compute, null_data == nullptr
- /// means all element need to do hash function, else only *null_data != 0 need to do hash func
- /// do xxHash here, faster than other hash method
- virtual void update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data = nullptr) const {
- LOG(FATAL) << get_name() << " update_hashes_with_value siphash not supported";
- }
-
/// Update state of hash function with value of n elements to avoid the virtual function call
/// null_data to mark whether need to do hash compute, null_data == nullptr
/// means all element need to do hash function, else only *null_data != 0 need to do hash func
diff --git a/be/src/vec/columns/column_array.cpp b/be/src/vec/columns/column_array.cpp
index 6d2914d8053..9abf13b9f3f 100644
--- a/be/src/vec/columns/column_array.cpp
+++ b/be/src/vec/columns/column_array.cpp
@@ -291,11 +291,6 @@ void ColumnArray::update_hash_with_value(size_t n, SipHash& hash) const {
for (size_t i = 0; i < array_size; ++i) get_data().update_hash_with_value(offset + i, hash);
}
-void ColumnArray::update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const {
- SIP_HASHES_FUNCTION_COLUMN_IMPL();
-}
-
// for every array row calculate xxHash
void ColumnArray::update_xxHash_with_value(size_t start, size_t end, uint64_t& hash,
const uint8_t* __restrict null_data) const {
diff --git a/be/src/vec/columns/column_array.h b/be/src/vec/columns/column_array.h
index 046bc22ac48..17408bfa633 100644
--- a/be/src/vec/columns/column_array.h
+++ b/be/src/vec/columns/column_array.h
@@ -142,9 +142,6 @@ public:
void update_crc_with_value(size_t start, size_t end, uint32_t& hash,
const uint8_t* __restrict null_data) const override;
- void update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const override;
-
void update_hashes_with_value(uint64_t* __restrict hashes,
const uint8_t* __restrict null_data = nullptr) const override;
diff --git a/be/src/vec/columns/column_complex.h b/be/src/vec/columns/column_complex.h
index 8e9686cb2a5..70d32da0b9b 100644
--- a/be/src/vec/columns/column_complex.h
+++ b/be/src/vec/columns/column_complex.h
@@ -220,15 +220,8 @@ public:
// TODO add hash function
}
- virtual void update_hashes_with_value(
- std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data = nullptr) const override {
- // TODO add hash function
- }
-
- virtual void update_hashes_with_value(
- uint64_t* __restrict hashes,
- const uint8_t* __restrict null_data = nullptr) const override {
+ void update_hashes_with_value(uint64_t* __restrict hashes,
+ const uint8_t* __restrict null_data = nullptr) const override {
// TODO add hash function
}
diff --git a/be/src/vec/columns/column_const.cpp b/be/src/vec/columns/column_const.cpp
index f7efec1f72a..e06e53b4289 100644
--- a/be/src/vec/columns/column_const.cpp
+++ b/be/src/vec/columns/column_const.cpp
@@ -93,22 +93,6 @@ ColumnPtr ColumnConst::permute(const Permutation& perm, size_t limit) const {
return ColumnConst::create(data, limit);
}
-void ColumnConst::update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const {
- DCHECK(null_data == nullptr);
- DCHECK(hashes.size() == size());
- auto real_data = data->get_data_at(0);
- if (real_data.data == nullptr) {
- for (auto& hash : hashes) {
- hash.update(0);
- }
- } else {
- for (auto& hash : hashes) {
- hash.update(real_data.data, real_data.size);
- }
- }
-}
-
void ColumnConst::update_crcs_with_value(uint32_t* __restrict hashes, doris::PrimitiveType type,
uint32_t rows, uint32_t offset,
const uint8_t* __restrict null_data) const {
diff --git a/be/src/vec/columns/column_const.h b/be/src/vec/columns/column_const.h
index 5498fbf7c20..746cb00fd5d 100644
--- a/be/src/vec/columns/column_const.h
+++ b/be/src/vec/columns/column_const.h
@@ -177,9 +177,6 @@ public:
data->update_hash_with_value(0, hash);
}
- void update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const override;
-
// (TODO.Amory) here may not use column_const update hash, and PrimitiveType is not used.
void update_crcs_with_value(uint32_t* __restrict hashes, PrimitiveType type, uint32_t rows,
uint32_t offset = 0,
diff --git a/be/src/vec/columns/column_decimal.cpp b/be/src/vec/columns/column_decimal.cpp
index 95b247fc668..be81a7ad4cb 100644
--- a/be/src/vec/columns/column_decimal.cpp
+++ b/be/src/vec/columns/column_decimal.cpp
@@ -131,12 +131,6 @@ void ColumnDecimal<T>::update_hash_with_value(size_t n, SipHash& hash) const {
hash.update(data[n]);
}
-template <typename T>
-void ColumnDecimal<T>::update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const {
- SIP_HASHES_FUNCTION_COLUMN_IMPL();
-}
-
template <typename T>
void ColumnDecimal<T>::update_crc_with_value(size_t start, size_t end, uint32_t& hash,
const uint8_t* __restrict null_data) const {
diff --git a/be/src/vec/columns/column_decimal.h b/be/src/vec/columns/column_decimal.h
index 49b58ebaa4f..920f6ad8438 100644
--- a/be/src/vec/columns/column_decimal.h
+++ b/be/src/vec/columns/column_decimal.h
@@ -176,8 +176,6 @@ public:
const uint8_t* null_map) override;
void update_hash_with_value(size_t n, SipHash& hash) const override;
- void update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const override;
void update_hashes_with_value(uint64_t* __restrict hashes,
const uint8_t* __restrict null_data) const override;
void update_crcs_with_value(uint32_t* __restrict hashes, PrimitiveType type, uint32_t rows,
diff --git a/be/src/vec/columns/column_map.cpp b/be/src/vec/columns/column_map.cpp
index dcfcdb46d70..4a1df8a6e26 100644
--- a/be/src/vec/columns/column_map.cpp
+++ b/be/src/vec/columns/column_map.cpp
@@ -271,11 +271,6 @@ void ColumnMap::update_hash_with_value(size_t n, SipHash& hash) const {
}
}
-void ColumnMap::update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const {
- SIP_HASHES_FUNCTION_COLUMN_IMPL();
-}
-
void ColumnMap::update_xxHash_with_value(size_t start, size_t end, uint64_t& hash,
const uint8_t* __restrict null_data) const {
auto& offsets = get_offsets();
diff --git a/be/src/vec/columns/column_map.h b/be/src/vec/columns/column_map.h
index 206660d6a06..0e6ad3c3d91 100644
--- a/be/src/vec/columns/column_map.h
+++ b/be/src/vec/columns/column_map.h
@@ -182,9 +182,6 @@ public:
void update_crc_with_value(size_t start, size_t end, uint32_t& hash,
const uint8_t* __restrict null_data) const override;
- void update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const override;
-
void update_hashes_with_value(uint64_t* __restrict hashes,
const uint8_t* __restrict null_data = nullptr) const override;
diff --git a/be/src/vec/columns/column_nullable.cpp b/be/src/vec/columns/column_nullable.cpp
index 0cd671eb110..e20b87af826 100644
--- a/be/src/vec/columns/column_nullable.cpp
+++ b/be/src/vec/columns/column_nullable.cpp
@@ -93,25 +93,6 @@ void ColumnNullable::update_hash_with_value(size_t n, SipHash& hash) const {
}
}
-void ColumnNullable::update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const {
- DCHECK(null_data == nullptr);
- auto s = hashes.size();
- DCHECK(s == size());
- const auto* __restrict real_null_data =
- assert_cast<const ColumnUInt8&>(*null_map).get_data().data();
- if (!has_null()) {
- nested_column->update_hashes_with_value(hashes, nullptr);
- } else {
- for (int i = 0; i < s; ++i) {
- if (real_null_data[i] != 0) {
- hashes[i].update(0);
- }
- }
- nested_column->update_hashes_with_value(hashes, real_null_data);
- }
-}
-
void ColumnNullable::update_crcs_with_value(uint32_t* __restrict hashes, doris::PrimitiveType type,
uint32_t rows, uint32_t offset,
const uint8_t* __restrict null_data) const {
diff --git a/be/src/vec/columns/column_nullable.h b/be/src/vec/columns/column_nullable.h
index eca4c57fceb..de01907650e 100644
--- a/be/src/vec/columns/column_nullable.h
+++ b/be/src/vec/columns/column_nullable.h
@@ -219,8 +219,6 @@ public:
const uint8_t* __restrict null_data) const override;
void update_hash_with_value(size_t n, SipHash& hash) const override;
- void update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const override;
void update_crcs_with_value(uint32_t* __restrict hash, PrimitiveType type, uint32_t rows,
uint32_t offset,
const uint8_t* __restrict null_data) const override;
diff --git a/be/src/vec/columns/column_string.h b/be/src/vec/columns/column_string.h
index 5c50f5ed2f1..9bf43c9c627 100644
--- a/be/src/vec/columns/column_string.h
+++ b/be/src/vec/columns/column_string.h
@@ -451,11 +451,6 @@ public:
hash.update(reinterpret_cast<const char*>(&chars[offset]), string_size);
}
- void update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const override {
- SIP_HASHES_FUNCTION_COLUMN_IMPL();
- }
-
void update_crcs_with_value(uint32_t* __restrict hashes, PrimitiveType type, uint32_t rows,
uint32_t offset,
const uint8_t* __restrict null_data) const override;
diff --git a/be/src/vec/columns/column_struct.cpp b/be/src/vec/columns/column_struct.cpp
index f6ab9c9604f..290452a1c8b 100644
--- a/be/src/vec/columns/column_struct.cpp
+++ b/be/src/vec/columns/column_struct.cpp
@@ -206,11 +206,6 @@ void ColumnStruct::update_hash_with_value(size_t n, SipHash& hash) const {
}
}
-void ColumnStruct::update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const {
- SIP_HASHES_FUNCTION_COLUMN_IMPL();
-}
-
void ColumnStruct::update_xxHash_with_value(size_t start, size_t end, uint64_t& hash,
const uint8_t* __restrict null_data) const {
for (const auto& column : columns) {
diff --git a/be/src/vec/columns/column_struct.h b/be/src/vec/columns/column_struct.h
index 1b1daee4452..d91d3800648 100644
--- a/be/src/vec/columns/column_struct.h
+++ b/be/src/vec/columns/column_struct.h
@@ -113,9 +113,6 @@ public:
void update_crc_with_value(size_t start, size_t end, uint32_t& hash,
const uint8_t* __restrict null_data) const override;
- void update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const override;
-
void update_hashes_with_value(uint64_t* __restrict hashes,
const uint8_t* __restrict null_data = nullptr) const override;
diff --git a/be/src/vec/columns/column_vector.cpp b/be/src/vec/columns/column_vector.cpp
index 71f55af0a79..acc6fc1d7f3 100644
--- a/be/src/vec/columns/column_vector.cpp
+++ b/be/src/vec/columns/column_vector.cpp
@@ -110,12 +110,6 @@ void ColumnVector<T>::update_hash_with_value(size_t n, SipHash& hash) const {
hash.update(data[n]);
}
-template <typename T>
-void ColumnVector<T>::update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const {
- SIP_HASHES_FUNCTION_COLUMN_IMPL();
-}
-
template <typename T>
void ColumnVector<T>::update_hashes_with_value(uint64_t* __restrict hashes,
const uint8_t* __restrict null_data) const {
diff --git a/be/src/vec/columns/column_vector.h b/be/src/vec/columns/column_vector.h
index dbc7524eaac..ff9197df357 100644
--- a/be/src/vec/columns/column_vector.h
+++ b/be/src/vec/columns/column_vector.h
@@ -321,9 +321,6 @@ public:
}
void update_hash_with_value(size_t n, SipHash& hash) const override;
- void update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const override;
-
void update_crcs_with_value(uint32_t* __restrict hashes, PrimitiveType type, uint32_t rows,
uint32_t offset,
const uint8_t* __restrict null_data) const override;
diff --git a/be/test/vec/columns/column_hash_func_test.cpp b/be/test/vec/columns/column_hash_func_test.cpp
index a95c7071af5..0a9471a1ab3 100644
--- a/be/test/vec/columns/column_hash_func_test.cpp
+++ b/be/test/vec/columns/column_hash_func_test.cpp
@@ -62,22 +62,14 @@ DataTypes create_scala_data_types() {
TEST(HashFuncTest, ArrayTypeTest) {
DataTypes dataTypes = create_scala_data_types();
- std::vector<uint64_t> sip_hash_vals(1);
std::vector<uint64_t> xx_hash_vals(1);
std::vector<uint32_t> crc_hash_vals(1);
- auto* __restrict sip_hashes = sip_hash_vals.data();
auto* __restrict xx_hashes = xx_hash_vals.data();
auto* __restrict crc_hashes = crc_hash_vals.data();
for (auto d : dataTypes) {
DataTypePtr a = std::make_shared<DataTypeArray>(d);
ColumnPtr col_a = a->create_column_const_with_default_value(1);
- // sipHash
- std::vector<SipHash> siphashs(1);
- col_a->update_hashes_with_value(siphashs);
- EXPECT_NO_FATAL_FAILURE(col_a->update_hashes_with_value(siphashs));
- sip_hashes[0] = siphashs[0].get64();
- std::cout << sip_hashes[0] << std::endl;
// xxHash
EXPECT_NO_FATAL_FAILURE(col_a->update_hashes_with_value(xx_hashes));
std::cout << xx_hashes[0] << std::endl;
@@ -184,22 +176,11 @@ TEST(HashFuncTest, ArrayCornerCaseTest) {
EXPECT_EQ(array_mutable_col->size(), 3);
- std::vector<uint64_t> sip_hash_vals(3);
std::vector<uint64_t> xx_hash_vals(3);
std::vector<uint32_t> crc_hash_vals(3);
- auto* __restrict sip_hashes = sip_hash_vals.data();
auto* __restrict xx_hashes = xx_hash_vals.data();
auto* __restrict crc_hashes = crc_hash_vals.data();
- // sipHash
- std::vector<SipHash> siphashs(3);
- array_mutable_col->update_hashes_with_value(siphashs);
- EXPECT_NO_FATAL_FAILURE(array_mutable_col->update_hashes_with_value(siphashs));
- sip_hashes[0] = siphashs[0].get64();
- sip_hashes[1] = siphashs[1].get64();
- sip_hashes[2] = siphashs[2].get64();
- EXPECT_EQ(sip_hashes[0], sip_hash_vals[1]);
- EXPECT_TRUE(sip_hash_vals[0] != sip_hash_vals[2]);
// xxHash
EXPECT_NO_FATAL_FAILURE(array_mutable_col->update_hashes_with_value(xx_hashes));
EXPECT_EQ(xx_hashes[0], xx_hashes[1]);
@@ -214,21 +195,14 @@ TEST(HashFuncTest, ArrayCornerCaseTest) {
TEST(HashFuncTest, MapTypeTest) {
DataTypes dataTypes = create_scala_data_types();
- std::vector<uint64_t> sip_hash_vals(1);
std::vector<uint64_t> xx_hash_vals(1);
std::vector<uint32_t> crc_hash_vals(1);
- auto* __restrict sip_hashes = sip_hash_vals.data();
auto* __restrict xx_hashes = xx_hash_vals.data();
auto* __restrict crc_hashes = crc_hash_vals.data();
// data_type_map
for (int i = 0; i < dataTypes.size() - 1; ++i) {
DataTypePtr a = std::make_shared<DataTypeMap>(dataTypes[i], dataTypes[i + 1]);
ColumnPtr col_a = a->create_column_const_with_default_value(1);
- // sipHash
- std::vector<SipHash> siphashs(1);
- EXPECT_NO_FATAL_FAILURE(unpack_if_const(col_a).first->update_hashes_with_value(siphashs));
- sip_hashes[0] = siphashs[0].get64();
- std::cout << sip_hashes[0] << std::endl;
// xxHash
EXPECT_NO_FATAL_FAILURE(unpack_if_const(col_a).first->update_hashes_with_value(xx_hashes));
std::cout << xx_hashes[0] << std::endl;
@@ -242,21 +216,14 @@ TEST(HashFuncTest, MapTypeTest) {
TEST(HashFuncTest, StructTypeTest) {
DataTypes dataTypes = create_scala_data_types();
- std::vector<uint64_t> sip_hash_vals(1);
std::vector<uint64_t> xx_hash_vals(1);
std::vector<uint32_t> crc_hash_vals(1);
- auto* __restrict sip_hashes = sip_hash_vals.data();
auto* __restrict xx_hashes = xx_hash_vals.data();
auto* __restrict crc_hashes = crc_hash_vals.data();
// data_type_struct
DataTypePtr a = std::make_shared<DataTypeStruct>(dataTypes);
ColumnPtr col_a = a->create_column_const_with_default_value(1);
- // sipHash
- std::vector<SipHash> siphashs(1);
- EXPECT_NO_FATAL_FAILURE(unpack_if_const(col_a).first->update_hashes_with_value(siphashs));
- sip_hashes[0] = siphashs[0].get64();
- std::cout << sip_hashes[0] << std::endl;
// xxHash
EXPECT_NO_FATAL_FAILURE(unpack_if_const(col_a).first->update_hashes_with_value(xx_hashes));
std::cout << xx_hashes[0] << std::endl;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 05/19: [regression test]Optimize testUniqueModelSchemaKeyChange (#31150)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 8a496bf554dc3ddb6ce1c78020f11d1a53a20605
Author: kkop <45...@users.noreply.github.com>
AuthorDate: Wed Feb 21 22:03:17 2024 +0800
[regression test]Optimize testUniqueModelSchemaKeyChange (#31150)
---
.../test_unique_model_schema_key_change.out | 121 +++++++++
.../test_unique_model_schema_key_change.groovy | 270 +++++++++++++++++++++
2 files changed, 391 insertions(+)
diff --git a/regression-test/data/schema_change_p0/test_unique_model_schema_key_change.out b/regression-test/data/schema_change_p0/test_unique_model_schema_key_change.out
new file mode 100644
index 00000000000..b8e5753aff9
--- /dev/null
+++ b/regression-test/data/schema_change_p0/test_unique_model_schema_key_change.out
@@ -0,0 +1,121 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- ! --
+123456789 Alice 广东省 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 广东省 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 广东省 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 广东省 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 广东省 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 广东省 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 广东省 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 四川省 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice false Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob false Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol false Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave false Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve false Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank false Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace false Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice false Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 0 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 0 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 0 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 0 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 0 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 0 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 0 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 0 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 999 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 999 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 999 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 999 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 999 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 999 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 999 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 567 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 999 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 999 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 999 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 999 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 999 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 999 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 999 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 2 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 99999991 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 99999991 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 99999991 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 99999991 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 99999991 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 99999991 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 99999991 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 88889494646 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 9999 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 9999 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 9999 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 9999 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 9999 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 9999 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 9999 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 555888555 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 16899.6464689000 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 16899.6464689000 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 16899.6464689000 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 16899.6464689000 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 16899.6464689000 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 16899.6464689000 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 16899.6464689000 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 16499.6464689000 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 1997-01-01 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1997-01-01 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1997-01-01 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1997-01-01 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 1997-01-01 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 1997-01-01 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1997-01-01 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 2024-01-01 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 1997-01-01T00:00 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1997-01-01T00:00 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1997-01-01T00:00 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1997-01-01T00:00 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 1997-01-01T00:00 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 1997-01-01T00:00 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1997-01-01T00:00 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 2024-01-04T09:00 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 1997-01-01T00:00 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1997-01-01T00:00 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1997-01-01T00:00 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1997-01-01T00:00 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 1997-01-01T00:00 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 1997-01-01T00:00 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1997-01-01T00:00 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 2024-01-04T09:00 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice F Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob F Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol F Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave F Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve F Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank F Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace F Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice T Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
diff --git a/regression-test/suites/schema_change_p0/test_unique_model_schema_key_change.groovy b/regression-test/suites/schema_change_p0/test_unique_model_schema_key_change.groovy
new file mode 100644
index 00000000000..deb094fc284
--- /dev/null
+++ b/regression-test/suites/schema_change_p0/test_unique_model_schema_key_change.groovy
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_unique_model_schema_key_change","p0") {
+ def tbName = "test_unique_model_schema_key_change"
+
+ //Test the unique model by adding a key column
+ sql """ DROP TABLE IF EXISTS ${tbName} """
+ def initTable = " CREATE TABLE IF NOT EXISTS ${tbName}\n" +
+ " (\n" +
+ " `user_id` LARGEINT NOT NULL COMMENT \"用户id\",\n" +
+ " `username` VARCHAR(50) NOT NULL COMMENT \"用户昵称\",\n" +
+ " `city` VARCHAR(20) COMMENT \"用户所在城市\",\n" +
+ " `age` SMALLINT COMMENT \"用户年龄\",\n" +
+ " `sex` TINYINT COMMENT \"用户性别\",\n" +
+ " `phone` LARGEINT COMMENT \"用户电话\",\n" +
+ " `address` VARCHAR(500) COMMENT \"用户地址\",\n" +
+ " `register_time` DATETIME COMMENT \"用户注册时间\"\n" +
+ " )\n" +
+ " UNIQUE KEY(`user_id`, `username`)\n" +
+ " DISTRIBUTED BY HASH(`user_id`) BUCKETS 1\n" +
+ " PROPERTIES (\n" +
+ " \"replication_allocation\" = \"tag.location.default: 1\",\n" +
+ " \"enable_unique_key_merge_on_write\" = \"true\"\n" +
+ " );"
+
+ def initTableData = "insert into ${tbName} values(123456789, 'Alice', 'Beijing', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00')," +
+ " (234567890, 'Bob', 'Shanghai', 30, 1, 13998765432, 'No. 456 Street, Shanghai', '2022-02-02 12:00:00')," +
+ " (345678901, 'Carol', 'Guangzhou', 28, 0, 13724681357, 'No. 789 Street, Guangzhou', '2022-03-03 14:00:00')," +
+ " (456789012, 'Dave', 'Shenzhen', 35, 1, 13680864279, 'No. 987 Street, Shenzhen', '2022-04-04 16:00:00')," +
+ " (567890123, 'Eve', 'Chengdu', 27, 0, 13572468091, 'No. 654 Street, Chengdu', '2022-05-05 18:00:00')," +
+ " (678901234, 'Frank', 'Hangzhou', 32, 1, 13467985213, 'No. 321 Street, Hangzhou', '2022-06-06 20:00:00')," +
+ " (789012345, 'Grace', 'Xian', 29, 0, 13333333333, 'No. 222 Street, Xian', '2022-07-07 22:00:00');"
+
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column province VARCHAR(20) KEY DEFAULT "广东省" AFTER username """
+ def insertSql = "insert into ${tbName} values(123456689, 'Alice', '四川省', 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00');"
+ def getTableStatusSql = " SHOW ALTER TABLE COLUMN WHERE IndexName='${tbName}' ORDER BY createtime DESC LIMIT 1 "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+ //Test the unique model by adding a key column with BOOLEAN
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column special_area BOOLEAN KEY DEFAULT "0" AFTER username """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 0, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+
+ //Test the unique model by adding a key column with TINYINT
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column special_area TINYINT KEY DEFAULT "0" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', 0, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+
+ //Test the unique model by adding a key column with SMALLINT
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column area_num SMALLINT KEY DEFAULT "999" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', 567, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+ //Test the unique model by adding a key column with INT
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column house_price INT KEY DEFAULT "999" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', 2, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+ //Test the unique model by adding a key column with BIGINT
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column house_price1 BIGINT KEY DEFAULT "99999991" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', 88889494646, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+
+ //Test the unique model by adding a key column with LARGEINT
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column car_price LARGEINT KEY DEFAULT "9999" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', 555888555, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00');"
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+
+
+ //Test the unique model by adding a key column with FLOAT
+ //java.sql.SQLException: errCode = 2, detailMessage = Float or double can not used as a key, use decimal instead.
+/* sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column phone FLOAT KEY DEFAULT "166.6" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', 189.9, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00');"
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")*/
+
+
+
+
+ //Test the unique model by adding a key column with DOUBLE
+ //java.sql.SQLException: errCode = 2, detailMessage = Float or double can not used as a key, use decimal instead.
+/* sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column watch FLOAT KEY DEFAULT "166.689" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', 189.479, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")*/
+
+
+
+
+ //Test the unique model by adding a key column with DECIMAL
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column watch DECIMAL(38,10) KEY DEFAULT "16899.6464689" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', 16499.6464689, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00');"
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+
+ //Test the unique model by adding a key column with DATE
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column watch DATE KEY DEFAULT "1997-01-01" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', \"2024-01-01\", 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+
+
+ //Test the unique model by adding a key column with DATETIME
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column anniversary DATETIME KEY DEFAULT "1997-01-01 00:00:00" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', \"2024-01-04 09:00:00\", 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+
+ //Test the unique model by adding a key column with DATETIME
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column anniversary DATETIME KEY DEFAULT "1997-01-01 00:00:00" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', \"2024-01-04 09:00:00\", 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+
+ //Test the unique model by adding a key column with CHAR
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column teacher CHAR KEY DEFAULT "F" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', \"T\", 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+
+ //Test the unique model by adding a key column with STRING
+ //java.sql.SQLException: errCode = 2, detailMessage = String Type should not be used in key column[comment].
+/* sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column comment STRING KEY DEFAULT "我是小说家" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', '我是侦探家', 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")*/
+
+
+ //Test the unique model by adding a key column with bitmap
+ //java.sql.SQLException: errCode = 2, detailMessage = Key column can not set complex type:device_id
+/* sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column device_id bitmap KEY DEFAULT "to_bitmap(243)" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', to_bitmap(243), 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")*/
+
+
+
+ //Test the unique model by adding a key column with Map
+ //java.sql.SQLException: errCode = 2, detailMessage = Map can only be used in the non-key column of the duplicate table at present.
+/* sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column m Map<STRING, INT> KEY DEFAULT "{'a': 100, 'b': 200}" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', '{'a': 100, 'b': 200}', 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")*/
+
+
+
+ //Test the unique model by adding a key column with JSON
+ //java.sql.SQLException: errCode = 2, detailMessage = JSONB type should not be used in key column[j].
+/* sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} add column j JSON KEY DEFAULT "{'a': 100, 'b': 200}" AFTER username """
+ insertSql = " insert into ${tbName} values(123456689, 'Alice', '{\"k1\":\"v31\", \"k2\": 300}', 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")*/
+
+
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 14/19: [pipelineX](refactor) Delete subclasses inherited from Dependency (#31216)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 52b9af06fb4b7ce3907cb2db0d4042f2bffc337d
Author: Gabriel <ga...@gmail.com>
AuthorDate: Thu Feb 22 10:01:23 2024 +0800
[pipelineX](refactor) Delete subclasses inherited from Dependency (#31216)
---
be/src/pipeline/exec/aggregation_sink_operator.h | 12 +-
be/src/pipeline/exec/aggregation_source_operator.h | 12 +-
be/src/pipeline/exec/analytic_sink_operator.cpp | 2 +-
be/src/pipeline/exec/analytic_sink_operator.h | 12 +-
be/src/pipeline/exec/analytic_source_operator.cpp | 6 +-
be/src/pipeline/exec/analytic_source_operator.h | 10 +-
be/src/pipeline/exec/assert_num_rows_operator.h | 4 +-
be/src/pipeline/exec/data_queue.h | 5 -
.../distinct_streaming_aggregation_operator.cpp | 2 +-
.../exec/distinct_streaming_aggregation_operator.h | 4 +-
be/src/pipeline/exec/empty_set_operator.h | 4 +-
be/src/pipeline/exec/es_scan_operator.cpp | 2 +-
be/src/pipeline/exec/exchange_sink_buffer.h | 5 +-
be/src/pipeline/exec/exchange_sink_operator.cpp | 5 +-
be/src/pipeline/exec/exchange_sink_operator.h | 62 +++-----
be/src/pipeline/exec/exchange_source_operator.cpp | 4 +-
be/src/pipeline/exec/exchange_source_operator.h | 26 +---
be/src/pipeline/exec/file_scan_operator.cpp | 2 +-
be/src/pipeline/exec/hashjoin_build_sink.cpp | 5 +-
be/src/pipeline/exec/hashjoin_build_sink.h | 14 +-
be/src/pipeline/exec/hashjoin_probe_operator.cpp | 4 +-
be/src/pipeline/exec/hashjoin_probe_operator.h | 10 +-
be/src/pipeline/exec/join_build_sink_operator.cpp | 20 +--
be/src/pipeline/exec/join_build_sink_operator.h | 6 +-
be/src/pipeline/exec/join_probe_operator.cpp | 24 +--
be/src/pipeline/exec/join_probe_operator.h | 8 +-
be/src/pipeline/exec/multi_cast_data_stream_sink.h | 12 +-
.../exec/multi_cast_data_stream_source.cpp | 3 +-
.../pipeline/exec/multi_cast_data_stream_source.h | 15 +-
be/src/pipeline/exec/multi_cast_data_streamer.h | 6 +-
.../exec/nested_loop_join_build_operator.cpp | 4 +-
.../exec/nested_loop_join_build_operator.h | 10 +-
.../exec/nested_loop_join_probe_operator.cpp | 6 +-
.../exec/nested_loop_join_probe_operator.h | 10 +-
be/src/pipeline/exec/olap_scan_operator.cpp | 4 +-
.../pipeline/exec/partition_sort_sink_operator.cpp | 2 +-
.../pipeline/exec/partition_sort_sink_operator.h | 12 +-
.../exec/partition_sort_source_operator.cpp | 2 +-
.../pipeline/exec/partition_sort_source_operator.h | 14 +-
be/src/pipeline/exec/repeat_operator.h | 4 +-
be/src/pipeline/exec/result_sink_operator.cpp | 3 +-
be/src/pipeline/exec/result_sink_operator.h | 13 +-
be/src/pipeline/exec/scan_operator.cpp | 24 ++-
be/src/pipeline/exec/scan_operator.h | 47 +-----
be/src/pipeline/exec/select_operator.h | 4 +-
be/src/pipeline/exec/set_probe_sink_operator.cpp | 5 +-
be/src/pipeline/exec/set_probe_sink_operator.h | 21 +--
be/src/pipeline/exec/set_sink_operator.cpp | 4 +-
be/src/pipeline/exec/set_sink_operator.h | 16 +-
be/src/pipeline/exec/set_source_operator.cpp | 2 +-
be/src/pipeline/exec/set_source_operator.h | 12 +-
be/src/pipeline/exec/sort_sink_operator.cpp | 2 +-
be/src/pipeline/exec/sort_sink_operator.h | 12 +-
be/src/pipeline/exec/sort_source_operator.cpp | 2 +-
be/src/pipeline/exec/sort_source_operator.h | 10 +-
.../pipeline/exec/streaming_aggregation_operator.h | 4 +-
be/src/pipeline/exec/union_sink_operator.h | 12 +-
be/src/pipeline/exec/union_source_operator.h | 12 +-
be/src/pipeline/pipeline_x/dependency.cpp | 2 +-
be/src/pipeline/pipeline_x/dependency.h | 46 ++++--
.../local_exchange/local_exchange_sink_operator.h | 13 +-
.../local_exchange_source_operator.cpp | 11 --
.../local_exchange_source_operator.h | 15 +-
.../pipeline_x/local_exchange/local_exchanger.h | 1 -
be/src/pipeline/pipeline_x/operator.cpp | 163 +++++++++++----------
be/src/pipeline/pipeline_x/operator.h | 36 ++---
.../pipeline_x/pipeline_x_fragment_context.cpp | 9 +-
.../pipeline_x/pipeline_x_fragment_context.h | 5 +-
be/src/pipeline/pipeline_x/pipeline_x_task.cpp | 14 +-
be/src/pipeline/pipeline_x/pipeline_x_task.h | 7 +-
be/src/runtime/buffer_control_block.cpp | 2 +-
be/src/runtime/buffer_control_block.h | 6 +-
be/src/vec/exec/scan/pip_scanner_context.h | 6 +-
be/src/vec/exec/scan/scanner_context.h | 1 -
be/src/vec/runtime/shared_hash_table_controller.h | 7 +-
be/src/vec/runtime/vdata_stream_recvr.cpp | 10 +-
be/src/vec/runtime/vdata_stream_recvr.h | 18 +--
be/src/vec/sink/vdata_stream_sender.cpp | 3 +-
be/src/vec/sink/vdata_stream_sender.h | 4 +-
79 files changed, 336 insertions(+), 622 deletions(-)
diff --git a/be/src/pipeline/exec/aggregation_sink_operator.h b/be/src/pipeline/exec/aggregation_sink_operator.h
index 2cbfe6c94e0..03f8dea22ae 100644
--- a/be/src/pipeline/exec/aggregation_sink_operator.h
+++ b/be/src/pipeline/exec/aggregation_sink_operator.h
@@ -45,20 +45,12 @@ public:
bool can_write() override { return true; }
};
-class AggSinkDependency final : public Dependency {
-public:
- using SharedState = AggSharedState;
- AggSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "AggSinkDependency", true, query_ctx) {}
- ~AggSinkDependency() override = default;
-};
-
class AggSinkOperatorX;
-class AggSinkLocalState : public PipelineXSinkLocalState<AggSinkDependency> {
+class AggSinkLocalState : public PipelineXSinkLocalState<AggSharedState> {
public:
ENABLE_FACTORY_CREATOR(AggSinkLocalState);
- using Base = PipelineXSinkLocalState<DependencyType>;
+ using Base = PipelineXSinkLocalState<AggSharedState>;
AggSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state);
~AggSinkLocalState() override = default;
diff --git a/be/src/pipeline/exec/aggregation_source_operator.h b/be/src/pipeline/exec/aggregation_source_operator.h
index d6aa19e57fd..08265dd8811 100644
--- a/be/src/pipeline/exec/aggregation_source_operator.h
+++ b/be/src/pipeline/exec/aggregation_source_operator.h
@@ -48,19 +48,11 @@ public:
Status open(RuntimeState*) override { return Status::OK(); }
};
-class AggSourceDependency final : public Dependency {
-public:
- using SharedState = AggSharedState;
- AggSourceDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "AggSourceDependency", query_ctx) {}
- ~AggSourceDependency() override = default;
-};
-
class AggSourceOperatorX;
-class AggLocalState final : public PipelineXLocalState<AggSourceDependency> {
+class AggLocalState final : public PipelineXLocalState<AggSharedState> {
public:
- using Base = PipelineXLocalState<AggSourceDependency>;
+ using Base = PipelineXLocalState<AggSharedState>;
ENABLE_FACTORY_CREATOR(AggLocalState);
AggLocalState(RuntimeState* state, OperatorXBase* parent);
~AggLocalState() override = default;
diff --git a/be/src/pipeline/exec/analytic_sink_operator.cpp b/be/src/pipeline/exec/analytic_sink_operator.cpp
index e9cc0cb1d06..0c00ab36d0c 100644
--- a/be/src/pipeline/exec/analytic_sink_operator.cpp
+++ b/be/src/pipeline/exec/analytic_sink_operator.cpp
@@ -27,7 +27,7 @@ namespace doris::pipeline {
OPERATOR_CODE_GENERATOR(AnalyticSinkOperator, StreamingOperator)
Status AnalyticSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& info) {
- RETURN_IF_ERROR(PipelineXSinkLocalState<AnalyticSinkDependency>::init(state, info));
+ RETURN_IF_ERROR(PipelineXSinkLocalState<AnalyticSharedState>::init(state, info));
SCOPED_TIMER(exec_time_counter());
SCOPED_TIMER(_open_timer);
auto& p = _parent->cast<AnalyticSinkOperatorX>();
diff --git a/be/src/pipeline/exec/analytic_sink_operator.h b/be/src/pipeline/exec/analytic_sink_operator.h
index c291aa95226..587fc2c9906 100644
--- a/be/src/pipeline/exec/analytic_sink_operator.h
+++ b/be/src/pipeline/exec/analytic_sink_operator.h
@@ -45,22 +45,14 @@ public:
bool can_write() override { return _node->can_write(); }
};
-class AnalyticSinkDependency final : public Dependency {
-public:
- using SharedState = AnalyticSharedState;
- AnalyticSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "AnalyticSinkDependency", true, query_ctx) {}
- ~AnalyticSinkDependency() override = default;
-};
-
class AnalyticSinkOperatorX;
-class AnalyticSinkLocalState : public PipelineXSinkLocalState<AnalyticSinkDependency> {
+class AnalyticSinkLocalState : public PipelineXSinkLocalState<AnalyticSharedState> {
ENABLE_FACTORY_CREATOR(AnalyticSinkLocalState);
public:
AnalyticSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
- : PipelineXSinkLocalState<AnalyticSinkDependency>(parent, state) {}
+ : PipelineXSinkLocalState<AnalyticSharedState>(parent, state) {}
Status init(RuntimeState* state, LocalSinkStateInfo& info) override;
diff --git a/be/src/pipeline/exec/analytic_source_operator.cpp b/be/src/pipeline/exec/analytic_source_operator.cpp
index f78030f7c17..49f03fd4936 100644
--- a/be/src/pipeline/exec/analytic_source_operator.cpp
+++ b/be/src/pipeline/exec/analytic_source_operator.cpp
@@ -27,7 +27,7 @@ namespace doris::pipeline {
OPERATOR_CODE_GENERATOR(AnalyticSourceOperator, SourceOperator)
AnalyticLocalState::AnalyticLocalState(RuntimeState* state, OperatorXBase* parent)
- : PipelineXLocalState<AnalyticSourceDependency>(state, parent),
+ : PipelineXLocalState<AnalyticSharedState>(state, parent),
_output_block_index(0),
_window_end_position(0),
_next_partition(false),
@@ -159,7 +159,7 @@ bool AnalyticLocalState::_whether_need_next_partition(
}
Status AnalyticLocalState::init(RuntimeState* state, LocalStateInfo& info) {
- RETURN_IF_ERROR(PipelineXLocalState<AnalyticSourceDependency>::init(state, info));
+ RETURN_IF_ERROR(PipelineXLocalState<AnalyticSharedState>::init(state, info));
SCOPED_TIMER(exec_time_counter());
SCOPED_TIMER(_open_timer);
_agg_arena_pool = std::make_unique<vectorized::Arena>();
@@ -564,7 +564,7 @@ Status AnalyticLocalState::close(RuntimeState* state) {
std::vector<vectorized::MutableColumnPtr> tmp_result_window_columns;
_result_window_columns.swap(tmp_result_window_columns);
- return PipelineXLocalState<AnalyticSourceDependency>::close(state);
+ return PipelineXLocalState<AnalyticSharedState>::close(state);
}
Status AnalyticSourceOperatorX::prepare(RuntimeState* state) {
diff --git a/be/src/pipeline/exec/analytic_source_operator.h b/be/src/pipeline/exec/analytic_source_operator.h
index b4acb2fbce5..b13ae6aab2b 100644
--- a/be/src/pipeline/exec/analytic_source_operator.h
+++ b/be/src/pipeline/exec/analytic_source_operator.h
@@ -46,16 +46,8 @@ public:
Status open(RuntimeState*) override { return Status::OK(); }
};
-class AnalyticSourceDependency final : public Dependency {
-public:
- using SharedState = AnalyticSharedState;
- AnalyticSourceDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "AnalyticSourceDependency", query_ctx) {}
- ~AnalyticSourceDependency() override = default;
-};
-
class AnalyticSourceOperatorX;
-class AnalyticLocalState final : public PipelineXLocalState<AnalyticSourceDependency> {
+class AnalyticLocalState final : public PipelineXLocalState<AnalyticSharedState> {
public:
ENABLE_FACTORY_CREATOR(AnalyticLocalState);
AnalyticLocalState(RuntimeState* state, OperatorXBase* parent);
diff --git a/be/src/pipeline/exec/assert_num_rows_operator.h b/be/src/pipeline/exec/assert_num_rows_operator.h
index 43036b597cf..278bc5ab45b 100644
--- a/be/src/pipeline/exec/assert_num_rows_operator.h
+++ b/be/src/pipeline/exec/assert_num_rows_operator.h
@@ -39,12 +39,12 @@ public:
: StreamingOperator(operator_builder, node) {}
};
-class AssertNumRowsLocalState final : public PipelineXLocalState<FakeDependency> {
+class AssertNumRowsLocalState final : public PipelineXLocalState<FakeSharedState> {
public:
ENABLE_FACTORY_CREATOR(AssertNumRowsLocalState);
AssertNumRowsLocalState(RuntimeState* state, OperatorXBase* parent)
- : PipelineXLocalState<FakeDependency>(state, parent) {}
+ : PipelineXLocalState<FakeSharedState>(state, parent) {}
~AssertNumRowsLocalState() = default;
};
diff --git a/be/src/pipeline/exec/data_queue.h b/be/src/pipeline/exec/data_queue.h
index 144fe62728c..1a50b7485d1 100644
--- a/be/src/pipeline/exec/data_queue.h
+++ b/be/src/pipeline/exec/data_queue.h
@@ -71,11 +71,6 @@ public:
void set_source_block();
private:
- friend class AggSourceDependency;
- friend class UnionSourceDependency;
- friend class AggSinkDependency;
- friend class UnionSinkDependency;
-
std::vector<std::unique_ptr<std::mutex>> _queue_blocks_lock;
std::vector<std::deque<std::unique_ptr<vectorized::Block>>> _queue_blocks;
diff --git a/be/src/pipeline/exec/distinct_streaming_aggregation_operator.cpp b/be/src/pipeline/exec/distinct_streaming_aggregation_operator.cpp
index 36d5065087a..66e2c333acd 100644
--- a/be/src/pipeline/exec/distinct_streaming_aggregation_operator.cpp
+++ b/be/src/pipeline/exec/distinct_streaming_aggregation_operator.cpp
@@ -33,7 +33,7 @@ namespace doris::pipeline {
DistinctStreamingAggLocalState::DistinctStreamingAggLocalState(RuntimeState* state,
OperatorXBase* parent)
- : PipelineXLocalState<FakeDependency>(state, parent),
+ : PipelineXLocalState<FakeSharedState>(state, parent),
dummy_mapped_data(std::make_shared<char>('A')),
_agg_arena_pool(std::make_unique<vectorized::Arena>()),
_agg_data(std::make_unique<vectorized::AggregatedDataVariants>()),
diff --git a/be/src/pipeline/exec/distinct_streaming_aggregation_operator.h b/be/src/pipeline/exec/distinct_streaming_aggregation_operator.h
index 564c29b3e2a..b35c3ed7e16 100644
--- a/be/src/pipeline/exec/distinct_streaming_aggregation_operator.h
+++ b/be/src/pipeline/exec/distinct_streaming_aggregation_operator.h
@@ -36,10 +36,10 @@ namespace pipeline {
class DistinctStreamingAggOperatorX;
-class DistinctStreamingAggLocalState final : public PipelineXLocalState<FakeDependency> {
+class DistinctStreamingAggLocalState final : public PipelineXLocalState<FakeSharedState> {
public:
using Parent = DistinctStreamingAggOperatorX;
- using Base = PipelineXLocalState<FakeDependency>;
+ using Base = PipelineXLocalState<FakeSharedState>;
ENABLE_FACTORY_CREATOR(DistinctStreamingAggLocalState);
DistinctStreamingAggLocalState(RuntimeState* state, OperatorXBase* parent);
diff --git a/be/src/pipeline/exec/empty_set_operator.h b/be/src/pipeline/exec/empty_set_operator.h
index 4e2607cd4ce..cd456265fa1 100644
--- a/be/src/pipeline/exec/empty_set_operator.h
+++ b/be/src/pipeline/exec/empty_set_operator.h
@@ -43,12 +43,12 @@ public:
bool can_read() override { return true; }
};
-class EmptySetLocalState final : public PipelineXLocalState<FakeDependency> {
+class EmptySetLocalState final : public PipelineXLocalState<FakeSharedState> {
public:
ENABLE_FACTORY_CREATOR(EmptySetLocalState);
EmptySetLocalState(RuntimeState* state, OperatorXBase* parent)
- : PipelineXLocalState<FakeDependency>(state, parent) {}
+ : PipelineXLocalState<FakeSharedState>(state, parent) {}
~EmptySetLocalState() = default;
};
diff --git a/be/src/pipeline/exec/es_scan_operator.cpp b/be/src/pipeline/exec/es_scan_operator.cpp
index c00ee6917ea..96283ac5052 100644
--- a/be/src/pipeline/exec/es_scan_operator.cpp
+++ b/be/src/pipeline/exec/es_scan_operator.cpp
@@ -67,7 +67,7 @@ Status EsScanLocalState::_process_conjuncts() {
Status EsScanLocalState::_init_scanners(std::list<vectorized::VScannerSPtr>* scanners) {
if (_scan_ranges.empty()) {
_eos = true;
- _scan_dependency->set_ready();
+ _dependency->set_ready();
return Status::OK();
}
diff --git a/be/src/pipeline/exec/exchange_sink_buffer.h b/be/src/pipeline/exec/exchange_sink_buffer.h
index 87fd378df4c..0afa59bf731 100644
--- a/be/src/pipeline/exec/exchange_sink_buffer.h
+++ b/be/src/pipeline/exec/exchange_sink_buffer.h
@@ -46,7 +46,6 @@ class TUniqueId;
using InstanceLoId = int64_t;
namespace pipeline {
-class ExchangeSinkQueueDependency;
class Dependency;
} // namespace pipeline
@@ -211,7 +210,7 @@ public:
void set_rpc_time(InstanceLoId id, int64_t start_rpc_time, int64_t receive_rpc_time);
void update_profile(RuntimeProfile* profile);
- void set_dependency(std::shared_ptr<ExchangeSinkQueueDependency> queue_dependency,
+ void set_dependency(std::shared_ptr<Dependency> queue_dependency,
std::shared_ptr<Dependency> finish_dependency) {
_queue_dependency = queue_dependency;
_finish_dependency = finish_dependency;
@@ -272,7 +271,7 @@ private:
std::atomic<int> _total_queue_size = 0;
static constexpr int QUEUE_CAPACITY_FACTOR = 64;
- std::shared_ptr<ExchangeSinkQueueDependency> _queue_dependency;
+ std::shared_ptr<Dependency> _queue_dependency;
std::shared_ptr<Dependency> _finish_dependency;
std::atomic<bool> _should_stop {false};
};
diff --git a/be/src/pipeline/exec/exchange_sink_operator.cpp b/be/src/pipeline/exec/exchange_sink_operator.cpp
index 9c26e975918..9f9a829da1d 100644
--- a/be/src/pipeline/exec/exchange_sink_operator.cpp
+++ b/be/src/pipeline/exec/exchange_sink_operator.cpp
@@ -171,8 +171,9 @@ Status ExchangeSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& inf
register_channels(_sink_buffer.get());
auto* _exchange_sink_dependency = _dependency;
- _queue_dependency = ExchangeSinkQueueDependency::create_shared(
- _parent->operator_id(), _parent->node_id(), state->get_query_ctx());
+ _queue_dependency =
+ Dependency::create_shared(_parent->operator_id(), _parent->node_id(),
+ "ExchangeSinkQueueDependency", true, state->get_query_ctx());
_sink_buffer->set_dependency(_queue_dependency, _finish_dependency);
_exchange_sink_dependency->add_child(_queue_dependency);
if ((p._part_type == TPartitionType::UNPARTITIONED || channels.size() == 1) &&
diff --git a/be/src/pipeline/exec/exchange_sink_operator.h b/be/src/pipeline/exec/exchange_sink_operator.h
index 5ba8228e502..9c258c15275 100644
--- a/be/src/pipeline/exec/exchange_sink_operator.h
+++ b/be/src/pipeline/exec/exchange_sink_operator.h
@@ -64,44 +64,9 @@ private:
int _mult_cast_id = -1;
};
-class ExchangeSinkQueueDependency final : public Dependency {
-public:
- ENABLE_FACTORY_CREATOR(ExchangeSinkQueueDependency);
- ExchangeSinkQueueDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "ResultQueueDependency", true, query_ctx) {}
- ~ExchangeSinkQueueDependency() override = default;
-};
-
-/**
- * We use this to control the execution for local exchange.
- * +---------------+ +---------------+ +---------------+
- * | ExchangeSink1 | | ExchangeSink2 | | ExchangeSink3 |
- * +---------------+ +---------------+ +---------------+
- * | | |
- * | +----------------------------+----------------------------------+ |
- * +----+------------------|------------------------------------------+ | |
- * | | +------------------------|--------------------|------------+-----+
- * Dependency 1-1 | Dependency 2-1 | Dependency 3-1 | Dependency 1-2 | Dependency 2-2 | Dependency 3-2 |
- * +----------------------------------------------+ +----------------------------------------------+
- * | queue1 queue2 queue3 | | queue1 queue2 queue3 |
- * | LocalRecvr | | LocalRecvr |
- * +----------------------------------------------+ +----------------------------------------------+
- * +-----------------+ +------------------+
- * | ExchangeSource1 | | ExchangeSource2 |
- * +-----------------+ +------------------+
- */
-class LocalExchangeChannelDependency final : public Dependency {
-public:
- ENABLE_FACTORY_CREATOR(LocalExchangeChannelDependency);
- LocalExchangeChannelDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "LocalExchangeChannelDependency", true, query_ctx) {}
- ~LocalExchangeChannelDependency() override = default;
- // TODO(gabriel): blocked by memory
-};
-
-class ExchangeSinkLocalState final : public PipelineXSinkLocalState<AndDependency> {
+class ExchangeSinkLocalState final : public PipelineXSinkLocalState<AndSharedState> {
ENABLE_FACTORY_CREATOR(ExchangeSinkLocalState);
- using Base = PipelineXSinkLocalState<AndDependency>;
+ using Base = PipelineXSinkLocalState<AndSharedState>;
public:
ExchangeSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
@@ -189,9 +154,28 @@ private:
vectorized::BlockSerializer<ExchangeSinkLocalState> _serializer;
- std::shared_ptr<ExchangeSinkQueueDependency> _queue_dependency;
+ std::shared_ptr<Dependency> _queue_dependency;
std::shared_ptr<Dependency> _broadcast_dependency;
- std::vector<std::shared_ptr<LocalExchangeChannelDependency>> _local_channels_dependency;
+
+ /**
+ * We use this to control the execution for local exchange.
+ * +---------------+ +---------------+ +---------------+
+ * | ExchangeSink1 | | ExchangeSink2 | | ExchangeSink3 |
+ * +---------------+ +---------------+ +---------------+
+ * | | |
+ * | +----------------------------+----------------------------------+ |
+ * +----+------------------|------------------------------------------+ | |
+ * | | +------------------------|--------------------|------------+-----+
+ * Dependency 1-1 | Dependency 2-1 | Dependency 3-1 | Dependency 1-2 | Dependency 2-2 | Dependency 3-2 |
+ * +----------------------------------------------+ +----------------------------------------------+
+ * | queue1 queue2 queue3 | | queue1 queue2 queue3 |
+ * | LocalRecvr | | LocalRecvr |
+ * +----------------------------------------------+ +----------------------------------------------+
+ * +-----------------+ +------------------+
+ * | ExchangeSource1 | | ExchangeSource2 |
+ * +-----------------+ +------------------+
+ */
+ std::vector<std::shared_ptr<Dependency>> _local_channels_dependency;
std::unique_ptr<vectorized::PartitionerBase> _partitioner;
int _partition_count;
diff --git a/be/src/pipeline/exec/exchange_source_operator.cpp b/be/src/pipeline/exec/exchange_source_operator.cpp
index 73aa7a22e81..22f171269d9 100644
--- a/be/src/pipeline/exec/exchange_source_operator.cpp
+++ b/be/src/pipeline/exec/exchange_source_operator.cpp
@@ -79,8 +79,8 @@ Status ExchangeLocalState::init(RuntimeState* state, LocalStateInfo& info) {
deps.resize(queues.size());
metrics.resize(queues.size());
for (size_t i = 0; i < queues.size(); i++) {
- deps[i] = ExchangeDataDependency::create_shared(_parent->operator_id(), _parent->node_id(),
- state->get_query_ctx(), queues[i]);
+ deps[i] = Dependency::create_shared(_parent->operator_id(), _parent->node_id(),
+ "SHUFFLE_DATA_DEPENDENCY", state->get_query_ctx());
queues[i]->set_dependency(deps[i]);
source_dependency->add_child(deps[i]);
}
diff --git a/be/src/pipeline/exec/exchange_source_operator.h b/be/src/pipeline/exec/exchange_source_operator.h
index 3340691f7ee..1ee68b90afb 100644
--- a/be/src/pipeline/exec/exchange_source_operator.h
+++ b/be/src/pipeline/exec/exchange_source_operator.h
@@ -50,32 +50,12 @@ public:
bool is_pending_finish() const override;
};
-struct ExchangeDataDependency final : public Dependency {
-public:
- ENABLE_FACTORY_CREATOR(ExchangeDataDependency);
- ExchangeDataDependency(int id, int node_id, QueryContext* query_ctx,
- vectorized::VDataStreamRecvr::SenderQueue* sender_queue)
- : Dependency(id, node_id, "DataDependency", query_ctx), _queue(sender_queue) {}
-
- std::string debug_string(int indentation_level) override {
- fmt::memory_buffer debug_string_buffer;
- fmt::format_to(debug_string_buffer,
- "{}, _is_cancelled = {}, _block_queue size = {},_num_remaining_senders = {}",
- Dependency::debug_string(indentation_level), _queue->_is_cancelled,
- _queue->_block_queue.size(), _queue->_num_remaining_senders);
- return fmt::to_string(debug_string_buffer);
- }
-
-private:
- vectorized::VDataStreamRecvr::SenderQueue* _queue;
-};
-
class ExchangeSourceOperatorX;
-class ExchangeLocalState final : public PipelineXLocalState<AndDependency> {
+class ExchangeLocalState final : public PipelineXLocalState<AndSharedState> {
ENABLE_FACTORY_CREATOR(ExchangeLocalState);
public:
- using Base = PipelineXLocalState<AndDependency>;
+ using Base = PipelineXLocalState<AndSharedState>;
ExchangeLocalState(RuntimeState* state, OperatorXBase* parent);
Status init(RuntimeState* state, LocalStateInfo& info) override;
@@ -87,7 +67,7 @@ public:
int64_t num_rows_skipped;
bool is_ready;
- std::vector<std::shared_ptr<ExchangeDataDependency>> deps;
+ std::vector<std::shared_ptr<Dependency>> deps;
std::vector<RuntimeProfile::Counter*> metrics;
};
diff --git a/be/src/pipeline/exec/file_scan_operator.cpp b/be/src/pipeline/exec/file_scan_operator.cpp
index ac193147dfb..ce0c5042cff 100644
--- a/be/src/pipeline/exec/file_scan_operator.cpp
+++ b/be/src/pipeline/exec/file_scan_operator.cpp
@@ -33,7 +33,7 @@ namespace doris::pipeline {
Status FileScanLocalState::_init_scanners(std::list<vectorized::VScannerSPtr>* scanners) {
if (_scan_ranges.empty()) {
_eos = true;
- _scan_dependency->set_ready();
+ _dependency->set_ready();
return Status::OK();
}
diff --git a/be/src/pipeline/exec/hashjoin_build_sink.cpp b/be/src/pipeline/exec/hashjoin_build_sink.cpp
index 9c6eff4cda8..0c3689b15b5 100644
--- a/be/src/pipeline/exec/hashjoin_build_sink.cpp
+++ b/be/src/pipeline/exec/hashjoin_build_sink.cpp
@@ -46,7 +46,6 @@ Status HashJoinBuildSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo
RETURN_IF_ERROR(JoinBuildSinkLocalState::init(state, info));
SCOPED_TIMER(exec_time_counter());
SCOPED_TIMER(_open_timer);
- _shared_hash_table_dependency = dependency_sptr();
auto& p = _parent->cast<HashJoinBuildSinkOperatorX>();
_shared_state->join_op_variants = p._join_op_variants;
@@ -73,9 +72,9 @@ Status HashJoinBuildSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo
}
}
if (!_should_build_hash_table) {
- _shared_hash_table_dependency->block();
+ _dependency->block();
p._shared_hashtable_controller->append_dependency(p.node_id(),
- _shared_hash_table_dependency);
+ _dependency->shared_from_this());
}
_build_blocks_memory_usage =
diff --git a/be/src/pipeline/exec/hashjoin_build_sink.h b/be/src/pipeline/exec/hashjoin_build_sink.h
index 134aba69a48..eb78d6bf515 100644
--- a/be/src/pipeline/exec/hashjoin_build_sink.h
+++ b/be/src/pipeline/exec/hashjoin_build_sink.h
@@ -45,20 +45,11 @@ public:
class HashJoinBuildSinkOperatorX;
-class SharedHashTableDependency final : public Dependency {
-public:
- using SharedState = HashJoinSharedState;
- ENABLE_FACTORY_CREATOR(SharedHashTableDependency);
- SharedHashTableDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "SharedHashTableBuildDependency", true, query_ctx) {}
- ~SharedHashTableDependency() override = default;
-};
-
class HashJoinBuildSinkLocalState final
- : public JoinBuildSinkLocalState<SharedHashTableDependency, HashJoinBuildSinkLocalState> {
+ : public JoinBuildSinkLocalState<HashJoinSharedState, HashJoinBuildSinkLocalState> {
public:
ENABLE_FACTORY_CREATOR(HashJoinBuildSinkLocalState);
- using Base = JoinBuildSinkLocalState<SharedHashTableDependency, HashJoinBuildSinkLocalState>;
+ using Base = JoinBuildSinkLocalState<HashJoinSharedState, HashJoinBuildSinkLocalState>;
using Parent = HashJoinBuildSinkOperatorX;
HashJoinBuildSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state);
~HashJoinBuildSinkLocalState() override = default;
@@ -114,7 +105,6 @@ protected:
* so null does not need to be added to the hash table.
*/
bool _build_side_ignore_null = false;
- std::shared_ptr<SharedHashTableDependency> _shared_hash_table_dependency;
std::vector<int> _build_col_ids;
RuntimeProfile::Counter* _build_table_timer = nullptr;
diff --git a/be/src/pipeline/exec/hashjoin_probe_operator.cpp b/be/src/pipeline/exec/hashjoin_probe_operator.cpp
index 1cf3e54572e..a0c4df528a7 100644
--- a/be/src/pipeline/exec/hashjoin_probe_operator.cpp
+++ b/be/src/pipeline/exec/hashjoin_probe_operator.cpp
@@ -28,7 +28,7 @@ namespace pipeline {
OPERATOR_CODE_GENERATOR(HashJoinProbeOperator, StatefulOperator)
HashJoinProbeLocalState::HashJoinProbeLocalState(RuntimeState* state, OperatorXBase* parent)
- : JoinProbeLocalState<HashJoinProbeDependency, HashJoinProbeLocalState>(state, parent) {}
+ : JoinProbeLocalState<HashJoinSharedState, HashJoinProbeLocalState>(state, parent) {}
Status HashJoinProbeLocalState::init(RuntimeState* state, LocalStateInfo& info) {
RETURN_IF_ERROR(JoinProbeLocalState::init(state, info));
@@ -147,7 +147,7 @@ Status HashJoinProbeLocalState::close(RuntimeState* state) {
_tuple_is_null_left_flag_column = nullptr;
_tuple_is_null_right_flag_column = nullptr;
_probe_block.clear();
- return JoinProbeLocalState<HashJoinProbeDependency, HashJoinProbeLocalState>::close(state);
+ return JoinProbeLocalState<HashJoinSharedState, HashJoinProbeLocalState>::close(state);
}
bool HashJoinProbeLocalState::_need_probe_null_map(vectorized::Block& block,
diff --git a/be/src/pipeline/exec/hashjoin_probe_operator.h b/be/src/pipeline/exec/hashjoin_probe_operator.h
index 4b7f9271920..43964442b03 100644
--- a/be/src/pipeline/exec/hashjoin_probe_operator.h
+++ b/be/src/pipeline/exec/hashjoin_probe_operator.h
@@ -63,17 +63,9 @@ using HashTableCtxVariants = std::variant<
vectorized::ProcessHashTableProbe<TJoinOp::NULL_AWARE_LEFT_SEMI_JOIN,
HashJoinProbeLocalState>>;
-class HashJoinProbeDependency final : public Dependency {
-public:
- using SharedState = HashJoinSharedState;
- HashJoinProbeDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "HashJoinProbeDependency", query_ctx) {}
- ~HashJoinProbeDependency() override = default;
-};
-
class HashJoinProbeOperatorX;
class HashJoinProbeLocalState final
- : public JoinProbeLocalState<HashJoinProbeDependency, HashJoinProbeLocalState> {
+ : public JoinProbeLocalState<HashJoinSharedState, HashJoinProbeLocalState> {
public:
using Parent = HashJoinProbeOperatorX;
ENABLE_FACTORY_CREATOR(HashJoinProbeLocalState);
diff --git a/be/src/pipeline/exec/join_build_sink_operator.cpp b/be/src/pipeline/exec/join_build_sink_operator.cpp
index 1141acc650d..ffef9d4dee0 100644
--- a/be/src/pipeline/exec/join_build_sink_operator.cpp
+++ b/be/src/pipeline/exec/join_build_sink_operator.cpp
@@ -23,21 +23,21 @@
namespace doris::pipeline {
-template <typename DependencyType, typename Derived>
-Status JoinBuildSinkLocalState<DependencyType, Derived>::init(RuntimeState* state,
+template <typename SharedStateArg, typename Derived>
+Status JoinBuildSinkLocalState<SharedStateArg, Derived>::init(RuntimeState* state,
LocalSinkStateInfo& info) {
- RETURN_IF_ERROR(PipelineXSinkLocalState<DependencyType>::init(state, info));
- auto& p = PipelineXSinkLocalState<DependencyType>::_parent
+ RETURN_IF_ERROR(PipelineXSinkLocalState<SharedStateArg>::init(state, info));
+ auto& p = PipelineXSinkLocalState<SharedStateArg>::_parent
->template cast<typename Derived::Parent>();
- PipelineXSinkLocalState<DependencyType>::profile()->add_info_string("JoinType",
+ PipelineXSinkLocalState<SharedStateArg>::profile()->add_info_string("JoinType",
to_string(p._join_op));
- _build_rows_counter = ADD_COUNTER(PipelineXSinkLocalState<DependencyType>::profile(),
+ _build_rows_counter = ADD_COUNTER(PipelineXSinkLocalState<SharedStateArg>::profile(),
"BuildRows", TUnit::UNIT);
- _publish_runtime_filter_timer = ADD_TIMER(PipelineXSinkLocalState<DependencyType>::profile(),
+ _publish_runtime_filter_timer = ADD_TIMER(PipelineXSinkLocalState<SharedStateArg>::profile(),
"PublishRuntimeFilterTime");
- _runtime_filter_compute_timer = ADD_TIMER(PipelineXSinkLocalState<DependencyType>::profile(),
+ _runtime_filter_compute_timer = ADD_TIMER(PipelineXSinkLocalState<SharedStateArg>::profile(),
"RuntimeFilterComputeTime");
return Status::OK();
@@ -122,9 +122,9 @@ void JoinBuildSinkOperatorX<LocalStateType>::_init_join_op() {
}
template class JoinBuildSinkOperatorX<HashJoinBuildSinkLocalState>;
-template class JoinBuildSinkLocalState<SharedHashTableDependency, HashJoinBuildSinkLocalState>;
+template class JoinBuildSinkLocalState<HashJoinSharedState, HashJoinBuildSinkLocalState>;
template class JoinBuildSinkOperatorX<NestedLoopJoinBuildSinkLocalState>;
-template class JoinBuildSinkLocalState<NestedLoopJoinBuildSinkDependency,
+template class JoinBuildSinkLocalState<NestedLoopJoinSharedState,
NestedLoopJoinBuildSinkLocalState>;
} // namespace doris::pipeline
diff --git a/be/src/pipeline/exec/join_build_sink_operator.h b/be/src/pipeline/exec/join_build_sink_operator.h
index 9cf5be80f80..f3d89976848 100644
--- a/be/src/pipeline/exec/join_build_sink_operator.h
+++ b/be/src/pipeline/exec/join_build_sink_operator.h
@@ -28,8 +28,8 @@ namespace pipeline {
template <typename LocalStateType>
class JoinBuildSinkOperatorX;
-template <typename DependencyType, typename Derived>
-class JoinBuildSinkLocalState : public PipelineXSinkLocalState<DependencyType> {
+template <typename SharedStateType, typename Derived>
+class JoinBuildSinkLocalState : public PipelineXSinkLocalState<SharedStateType> {
public:
Status init(RuntimeState* state, LocalSinkStateInfo& info) override;
@@ -37,7 +37,7 @@ public:
protected:
JoinBuildSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
- : PipelineXSinkLocalState<DependencyType>(parent, state) {}
+ : PipelineXSinkLocalState<SharedStateType>(parent, state) {}
~JoinBuildSinkLocalState() override = default;
template <typename LocalStateType>
friend class JoinBuildSinkOperatorX;
diff --git a/be/src/pipeline/exec/join_probe_operator.cpp b/be/src/pipeline/exec/join_probe_operator.cpp
index c041fed5d7a..469ceeaff62 100644
--- a/be/src/pipeline/exec/join_probe_operator.cpp
+++ b/be/src/pipeline/exec/join_probe_operator.cpp
@@ -23,8 +23,8 @@
namespace doris::pipeline {
-template <typename DependencyType, typename Derived>
-Status JoinProbeLocalState<DependencyType, Derived>::init(RuntimeState* state,
+template <typename SharedStateArg, typename Derived>
+Status JoinProbeLocalState<SharedStateArg, Derived>::init(RuntimeState* state,
LocalStateInfo& info) {
RETURN_IF_ERROR(Base::init(state, info));
auto& p = Base::_parent->template cast<typename Derived::Parent>();
@@ -46,8 +46,8 @@ Status JoinProbeLocalState<DependencyType, Derived>::init(RuntimeState* state,
return Status::OK();
}
-template <typename DependencyType, typename Derived>
-Status JoinProbeLocalState<DependencyType, Derived>::close(RuntimeState* state) {
+template <typename SharedStateArg, typename Derived>
+Status JoinProbeLocalState<SharedStateArg, Derived>::close(RuntimeState* state) {
if (Base::_closed) {
return Status::OK();
}
@@ -55,8 +55,8 @@ Status JoinProbeLocalState<DependencyType, Derived>::close(RuntimeState* state)
return Base::close(state);
}
-template <typename DependencyType, typename Derived>
-void JoinProbeLocalState<DependencyType, Derived>::_construct_mutable_join_block() {
+template <typename SharedStateArg, typename Derived>
+void JoinProbeLocalState<SharedStateArg, Derived>::_construct_mutable_join_block() {
auto& p = Base::_parent->template cast<typename Derived::Parent>();
const auto& mutable_block_desc = p._intermediate_row_desc;
for (const auto tuple_desc : mutable_block_desc->tuple_descriptors()) {
@@ -77,8 +77,8 @@ void JoinProbeLocalState<DependencyType, Derived>::_construct_mutable_join_block
}
}
-template <typename DependencyType, typename Derived>
-Status JoinProbeLocalState<DependencyType, Derived>::_build_output_block(
+template <typename SharedStateArg, typename Derived>
+Status JoinProbeLocalState<SharedStateArg, Derived>::_build_output_block(
vectorized::Block* origin_block, vectorized::Block* output_block, bool keep_origin) {
auto& p = Base::_parent->template cast<typename Derived::Parent>();
SCOPED_TIMER(_build_output_block_timer);
@@ -150,8 +150,8 @@ Status JoinProbeLocalState<DependencyType, Derived>::_build_output_block(
return Status::OK();
}
-template <typename DependencyType, typename Derived>
-void JoinProbeLocalState<DependencyType, Derived>::_reset_tuple_is_null_column() {
+template <typename SharedStateArg, typename Derived>
+void JoinProbeLocalState<SharedStateArg, Derived>::_reset_tuple_is_null_column() {
if (Base::_parent->template cast<typename Derived::Parent>()._is_outer_join) {
reinterpret_cast<vectorized::ColumnUInt8&>(*_tuple_is_null_left_flag_column).clear();
reinterpret_cast<vectorized::ColumnUInt8&>(*_tuple_is_null_right_flag_column).clear();
@@ -233,10 +233,10 @@ Status JoinProbeOperatorX<LocalStateType>::open(doris::RuntimeState* state) {
return vectorized::VExpr::open(_output_expr_ctxs, state);
}
-template class JoinProbeLocalState<HashJoinProbeDependency, HashJoinProbeLocalState>;
+template class JoinProbeLocalState<HashJoinSharedState, HashJoinProbeLocalState>;
template class JoinProbeOperatorX<HashJoinProbeLocalState>;
-template class JoinProbeLocalState<NestedLoopJoinProbeDependency, NestedLoopJoinProbeLocalState>;
+template class JoinProbeLocalState<NestedLoopJoinSharedState, NestedLoopJoinProbeLocalState>;
template class JoinProbeOperatorX<NestedLoopJoinProbeLocalState>;
} // namespace doris::pipeline
diff --git a/be/src/pipeline/exec/join_probe_operator.h b/be/src/pipeline/exec/join_probe_operator.h
index 6f379bc9749..eb2136d74b7 100644
--- a/be/src/pipeline/exec/join_probe_operator.h
+++ b/be/src/pipeline/exec/join_probe_operator.h
@@ -27,10 +27,10 @@ namespace doris {
namespace pipeline {
template <typename LocalStateType>
class JoinProbeOperatorX;
-template <typename DependencyType, typename Derived>
-class JoinProbeLocalState : public PipelineXLocalState<DependencyType> {
+template <typename SharedStateArg, typename Derived>
+class JoinProbeLocalState : public PipelineXLocalState<SharedStateArg> {
public:
- using Base = PipelineXLocalState<DependencyType>;
+ using Base = PipelineXLocalState<SharedStateArg>;
Status init(RuntimeState* state, LocalStateInfo& info) override;
Status close(RuntimeState* state) override;
virtual void add_tuple_is_null_column(vectorized::Block* block) = 0;
@@ -97,7 +97,7 @@ public:
}
protected:
- template <typename DependencyType, typename Derived>
+ template <typename SharedStateArg, typename Derived>
friend class JoinProbeLocalState;
const TJoinOp::type _join_op;
diff --git a/be/src/pipeline/exec/multi_cast_data_stream_sink.h b/be/src/pipeline/exec/multi_cast_data_stream_sink.h
index 7203ed0bf6f..d3159d5b460 100644
--- a/be/src/pipeline/exec/multi_cast_data_stream_sink.h
+++ b/be/src/pipeline/exec/multi_cast_data_stream_sink.h
@@ -41,23 +41,15 @@ public:
bool can_write() override { return _sink->can_write(); }
};
-class MultiCastSinkDependency final : public Dependency {
-public:
- using SharedState = MultiCastSharedState;
- MultiCastSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "MultiCastSinkDependency", true, query_ctx) {}
- ~MultiCastSinkDependency() override = default;
-};
-
class MultiCastDataStreamSinkOperatorX;
class MultiCastDataStreamSinkLocalState final
- : public PipelineXSinkLocalState<MultiCastSinkDependency> {
+ : public PipelineXSinkLocalState<MultiCastSharedState> {
ENABLE_FACTORY_CREATOR(MultiCastDataStreamSinkLocalState);
MultiCastDataStreamSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
: Base(parent, state) {}
friend class MultiCastDataStreamSinkOperatorX;
friend class DataSinkOperatorX<MultiCastDataStreamSinkLocalState>;
- using Base = PipelineXSinkLocalState<MultiCastSinkDependency>;
+ using Base = PipelineXSinkLocalState<MultiCastSharedState>;
using Parent = MultiCastDataStreamSinkOperatorX;
std::string name_suffix() override;
diff --git a/be/src/pipeline/exec/multi_cast_data_stream_source.cpp b/be/src/pipeline/exec/multi_cast_data_stream_source.cpp
index ccc0169c289..2ba0c2bd0ce 100644
--- a/be/src/pipeline/exec/multi_cast_data_stream_source.cpp
+++ b/be/src/pipeline/exec/multi_cast_data_stream_source.cpp
@@ -140,8 +140,7 @@ Status MultiCastDataStreamSourceLocalState::init(RuntimeState* state, LocalState
SCOPED_TIMER(exec_time_counter());
SCOPED_TIMER(_open_timer);
auto& p = _parent->cast<Parent>();
- _shared_state->multi_cast_data_streamer.set_dep_by_sender_idx(
- p._consumer_id, static_cast<MultiCastSourceDependency*>(_dependency));
+ _shared_state->multi_cast_data_streamer.set_dep_by_sender_idx(p._consumer_id, _dependency);
_output_expr_contexts.resize(p._output_expr_contexts.size());
for (size_t i = 0; i < p._output_expr_contexts.size(); i++) {
RETURN_IF_ERROR(p._output_expr_contexts[i]->clone(state, _output_expr_contexts[i]));
diff --git a/be/src/pipeline/exec/multi_cast_data_stream_source.h b/be/src/pipeline/exec/multi_cast_data_stream_source.h
index 8eaa82520d1..9f44790f4e8 100644
--- a/be/src/pipeline/exec/multi_cast_data_stream_source.h
+++ b/be/src/pipeline/exec/multi_cast_data_stream_source.h
@@ -92,22 +92,13 @@ private:
vectorized::VExprContextSPtrs _conjuncts;
};
-class MultiCastSourceDependency final : public Dependency {
-public:
- using SharedState = MultiCastSharedState;
- MultiCastSourceDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "MultiCastSourceDependency", query_ctx) {}
- ~MultiCastSourceDependency() override = default;
-};
-
class MultiCastDataStreamerSourceOperatorX;
-class MultiCastDataStreamSourceLocalState final
- : public PipelineXLocalState<MultiCastSourceDependency>,
- public vectorized::RuntimeFilterConsumer {
+class MultiCastDataStreamSourceLocalState final : public PipelineXLocalState<MultiCastSharedState>,
+ public vectorized::RuntimeFilterConsumer {
public:
ENABLE_FACTORY_CREATOR(MultiCastDataStreamSourceLocalState);
- using Base = PipelineXLocalState<MultiCastSourceDependency>;
+ using Base = PipelineXLocalState<MultiCastSharedState>;
using Parent = MultiCastDataStreamerSourceOperatorX;
MultiCastDataStreamSourceLocalState(RuntimeState* state, OperatorXBase* parent);
Status init(RuntimeState* state, LocalStateInfo& info) override;
diff --git a/be/src/pipeline/exec/multi_cast_data_streamer.h b/be/src/pipeline/exec/multi_cast_data_streamer.h
index 5e4179e0cad..2078a729227 100644
--- a/be/src/pipeline/exec/multi_cast_data_streamer.h
+++ b/be/src/pipeline/exec/multi_cast_data_streamer.h
@@ -21,7 +21,7 @@
namespace doris::pipeline {
-class MultiCastSourceDependency;
+class Dependency;
struct MultiCastBlock {
MultiCastBlock(vectorized::Block* block, int used_count, size_t mem_size);
@@ -74,7 +74,7 @@ public:
_set_ready_for_read();
}
- void set_dep_by_sender_idx(int sender_idx, MultiCastSourceDependency* dep) {
+ void set_dep_by_sender_idx(int sender_idx, Dependency* dep) {
_dependencies[sender_idx] = dep;
_block_reading(sender_idx);
}
@@ -97,6 +97,6 @@ private:
RuntimeProfile::Counter* _process_rows = nullptr;
RuntimeProfile::Counter* _peak_mem_usage = nullptr;
- std::vector<MultiCastSourceDependency*> _dependencies;
+ std::vector<Dependency*> _dependencies;
};
} // namespace doris::pipeline
\ No newline at end of file
diff --git a/be/src/pipeline/exec/nested_loop_join_build_operator.cpp b/be/src/pipeline/exec/nested_loop_join_build_operator.cpp
index 8485e2c0b24..a31c8cd4913 100644
--- a/be/src/pipeline/exec/nested_loop_join_build_operator.cpp
+++ b/be/src/pipeline/exec/nested_loop_join_build_operator.cpp
@@ -27,8 +27,8 @@ OPERATOR_CODE_GENERATOR(NestLoopJoinBuildOperator, StreamingOperator)
NestedLoopJoinBuildSinkLocalState::NestedLoopJoinBuildSinkLocalState(DataSinkOperatorXBase* parent,
RuntimeState* state)
- : JoinBuildSinkLocalState<NestedLoopJoinBuildSinkDependency,
- NestedLoopJoinBuildSinkLocalState>(parent, state) {}
+ : JoinBuildSinkLocalState<NestedLoopJoinSharedState, NestedLoopJoinBuildSinkLocalState>(
+ parent, state) {}
Status NestedLoopJoinBuildSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& info) {
RETURN_IF_ERROR(JoinBuildSinkLocalState::init(state, info));
diff --git a/be/src/pipeline/exec/nested_loop_join_build_operator.h b/be/src/pipeline/exec/nested_loop_join_build_operator.h
index e89d6a9d0be..14694889e1d 100644
--- a/be/src/pipeline/exec/nested_loop_join_build_operator.h
+++ b/be/src/pipeline/exec/nested_loop_join_build_operator.h
@@ -44,18 +44,10 @@ public:
bool can_write() override { return true; }
};
-class NestedLoopJoinBuildSinkDependency final : public Dependency {
-public:
- using SharedState = NestedLoopJoinSharedState;
- NestedLoopJoinBuildSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "NestedLoopJoinBuildSinkDependency", true, query_ctx) {}
- ~NestedLoopJoinBuildSinkDependency() override = default;
-};
-
class NestedLoopJoinBuildSinkOperatorX;
class NestedLoopJoinBuildSinkLocalState final
- : public JoinBuildSinkLocalState<NestedLoopJoinBuildSinkDependency,
+ : public JoinBuildSinkLocalState<NestedLoopJoinSharedState,
NestedLoopJoinBuildSinkLocalState> {
public:
ENABLE_FACTORY_CREATOR(NestedLoopJoinBuildSinkLocalState);
diff --git a/be/src/pipeline/exec/nested_loop_join_probe_operator.cpp b/be/src/pipeline/exec/nested_loop_join_probe_operator.cpp
index 1e3de36e920..8287219dfb2 100644
--- a/be/src/pipeline/exec/nested_loop_join_probe_operator.cpp
+++ b/be/src/pipeline/exec/nested_loop_join_probe_operator.cpp
@@ -44,8 +44,8 @@ Status NestLoopJoinProbeOperator::close(doris::RuntimeState* state) {
NestedLoopJoinProbeLocalState::NestedLoopJoinProbeLocalState(RuntimeState* state,
OperatorXBase* parent)
- : JoinProbeLocalState<NestedLoopJoinProbeDependency, NestedLoopJoinProbeLocalState>(state,
- parent),
+ : JoinProbeLocalState<NestedLoopJoinSharedState, NestedLoopJoinProbeLocalState>(state,
+ parent),
_matched_rows_done(false),
_left_block_pos(0) {}
@@ -74,7 +74,7 @@ Status NestedLoopJoinProbeLocalState::close(RuntimeState* state) {
_tuple_is_null_left_flag_column = nullptr;
_tuple_is_null_right_flag_column = nullptr;
- return JoinProbeLocalState<NestedLoopJoinProbeDependency, NestedLoopJoinProbeLocalState>::close(
+ return JoinProbeLocalState<NestedLoopJoinSharedState, NestedLoopJoinProbeLocalState>::close(
state);
}
diff --git a/be/src/pipeline/exec/nested_loop_join_probe_operator.h b/be/src/pipeline/exec/nested_loop_join_probe_operator.h
index 35302434e70..f1ba439bf19 100644
--- a/be/src/pipeline/exec/nested_loop_join_probe_operator.h
+++ b/be/src/pipeline/exec/nested_loop_join_probe_operator.h
@@ -51,17 +51,9 @@ public:
Status close(RuntimeState* state) override;
};
-class NestedLoopJoinProbeDependency final : public Dependency {
-public:
- using SharedState = NestedLoopJoinSharedState;
- NestedLoopJoinProbeDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "NestedLoopJoinProbeDependency", query_ctx) {}
- ~NestedLoopJoinProbeDependency() override = default;
-};
-
class NestedLoopJoinProbeOperatorX;
class NestedLoopJoinProbeLocalState final
- : public JoinProbeLocalState<NestedLoopJoinProbeDependency, NestedLoopJoinProbeLocalState> {
+ : public JoinProbeLocalState<NestedLoopJoinSharedState, NestedLoopJoinProbeLocalState> {
public:
using Parent = NestedLoopJoinProbeOperatorX;
ENABLE_FACTORY_CREATOR(NestedLoopJoinProbeLocalState);
diff --git a/be/src/pipeline/exec/olap_scan_operator.cpp b/be/src/pipeline/exec/olap_scan_operator.cpp
index 0aab714449e..f443cacf040 100644
--- a/be/src/pipeline/exec/olap_scan_operator.cpp
+++ b/be/src/pipeline/exec/olap_scan_operator.cpp
@@ -225,7 +225,7 @@ bool OlapScanLocalState::_storage_no_merge() {
Status OlapScanLocalState::_init_scanners(std::list<vectorized::VScannerSPtr>* scanners) {
if (_scan_ranges.empty()) {
_eos = true;
- _scan_dependency->set_ready();
+ _dependency->set_ready();
return Status::OK();
}
SCOPED_TIMER(_scanner_init_timer);
@@ -486,7 +486,7 @@ Status OlapScanLocalState::_build_key_ranges_and_filters() {
}
if (eos) {
_eos = true;
- _scan_dependency->set_ready();
+ _dependency->set_ready();
}
for (auto& iter : _colname_to_value_range) {
diff --git a/be/src/pipeline/exec/partition_sort_sink_operator.cpp b/be/src/pipeline/exec/partition_sort_sink_operator.cpp
index 6cdfca18d8b..03aa3d06f66 100644
--- a/be/src/pipeline/exec/partition_sort_sink_operator.cpp
+++ b/be/src/pipeline/exec/partition_sort_sink_operator.cpp
@@ -28,7 +28,7 @@ OperatorPtr PartitionSortSinkOperatorBuilder::build_operator() {
}
Status PartitionSortSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& info) {
- RETURN_IF_ERROR(PipelineXSinkLocalState<PartitionSortSinkDependency>::init(state, info));
+ RETURN_IF_ERROR(PipelineXSinkLocalState<PartitionSortNodeSharedState>::init(state, info));
SCOPED_TIMER(exec_time_counter());
auto& p = _parent->cast<PartitionSortSinkOperatorX>();
RETURN_IF_ERROR(p._vsort_exec_exprs.clone(state, _vsort_exec_exprs));
diff --git a/be/src/pipeline/exec/partition_sort_sink_operator.h b/be/src/pipeline/exec/partition_sort_sink_operator.h
index 7cb881e39ca..9dbb215b78e 100644
--- a/be/src/pipeline/exec/partition_sort_sink_operator.h
+++ b/be/src/pipeline/exec/partition_sort_sink_operator.h
@@ -50,21 +50,13 @@ public:
bool can_write() override { return true; }
};
-class PartitionSortSinkDependency final : public Dependency {
-public:
- using SharedState = PartitionSortNodeSharedState;
- PartitionSortSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "PartitionSortSinkDependency", true, query_ctx) {}
- ~PartitionSortSinkDependency() override = default;
-};
-
class PartitionSortSinkOperatorX;
-class PartitionSortSinkLocalState : public PipelineXSinkLocalState<PartitionSortSinkDependency> {
+class PartitionSortSinkLocalState : public PipelineXSinkLocalState<PartitionSortNodeSharedState> {
ENABLE_FACTORY_CREATOR(PartitionSortSinkLocalState);
public:
PartitionSortSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
- : PipelineXSinkLocalState<PartitionSortSinkDependency>(parent, state) {}
+ : PipelineXSinkLocalState<PartitionSortNodeSharedState>(parent, state) {}
Status init(RuntimeState* state, LocalSinkStateInfo& info) override;
diff --git a/be/src/pipeline/exec/partition_sort_source_operator.cpp b/be/src/pipeline/exec/partition_sort_source_operator.cpp
index c66ad2fc30a..b69a5d1873b 100644
--- a/be/src/pipeline/exec/partition_sort_source_operator.cpp
+++ b/be/src/pipeline/exec/partition_sort_source_operator.cpp
@@ -30,7 +30,7 @@ OperatorPtr PartitionSortSourceOperatorBuilder::build_operator() {
}
Status PartitionSortSourceLocalState::init(RuntimeState* state, LocalStateInfo& info) {
- RETURN_IF_ERROR(PipelineXLocalState<PartitionSortSourceDependency>::init(state, info));
+ RETURN_IF_ERROR(PipelineXLocalState<PartitionSortNodeSharedState>::init(state, info));
SCOPED_TIMER(exec_time_counter());
SCOPED_TIMER(_open_timer);
_get_sorted_timer = ADD_TIMER(profile(), "GetSortedTime");
diff --git a/be/src/pipeline/exec/partition_sort_source_operator.h b/be/src/pipeline/exec/partition_sort_source_operator.h
index d0ae26c7c4a..3fb544316a2 100644
--- a/be/src/pipeline/exec/partition_sort_source_operator.h
+++ b/be/src/pipeline/exec/partition_sort_source_operator.h
@@ -48,22 +48,14 @@ public:
Status open(RuntimeState*) override { return Status::OK(); }
};
-class PartitionSortSourceDependency final : public Dependency {
-public:
- using SharedState = PartitionSortNodeSharedState;
- PartitionSortSourceDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "PartitionSortSourceDependency", query_ctx) {}
- ~PartitionSortSourceDependency() override = default;
-};
-
class PartitionSortSourceOperatorX;
class PartitionSortSourceLocalState final
- : public PipelineXLocalState<PartitionSortSourceDependency> {
+ : public PipelineXLocalState<PartitionSortNodeSharedState> {
public:
ENABLE_FACTORY_CREATOR(PartitionSortSourceLocalState);
- using Base = PipelineXLocalState<PartitionSortSourceDependency>;
+ using Base = PipelineXLocalState<PartitionSortNodeSharedState>;
PartitionSortSourceLocalState(RuntimeState* state, OperatorXBase* parent)
- : PipelineXLocalState<PartitionSortSourceDependency>(state, parent),
+ : PipelineXLocalState<PartitionSortNodeSharedState>(state, parent),
_get_sorted_timer(nullptr) {}
Status init(RuntimeState* state, LocalStateInfo& info) override;
diff --git a/be/src/pipeline/exec/repeat_operator.h b/be/src/pipeline/exec/repeat_operator.h
index 00555ce9dcf..768424722a8 100644
--- a/be/src/pipeline/exec/repeat_operator.h
+++ b/be/src/pipeline/exec/repeat_operator.h
@@ -47,11 +47,11 @@ public:
class RepeatOperatorX;
-class RepeatLocalState final : public PipelineXLocalState<FakeDependency> {
+class RepeatLocalState final : public PipelineXLocalState<FakeSharedState> {
public:
ENABLE_FACTORY_CREATOR(RepeatLocalState);
using Parent = RepeatOperatorX;
- using Base = PipelineXLocalState<FakeDependency>;
+ using Base = PipelineXLocalState<FakeSharedState>;
RepeatLocalState(RuntimeState* state, OperatorXBase* parent);
Status init(RuntimeState* state, LocalStateInfo& info) override;
diff --git a/be/src/pipeline/exec/result_sink_operator.cpp b/be/src/pipeline/exec/result_sink_operator.cpp
index 8b3afb1908f..7eb7cd7ef86 100644
--- a/be/src/pipeline/exec/result_sink_operator.cpp
+++ b/be/src/pipeline/exec/result_sink_operator.cpp
@@ -62,10 +62,9 @@ Status ResultSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& info)
RETURN_IF_ERROR(state->exec_env()->result_mgr()->create_sender(
state->fragment_instance_id(), vectorized::RESULT_SINK_BUFFER_SIZE, &_sender, true,
state->execution_timeout()));
- auto result_sink_dependency = dependency_sptr();
_blocks_sent_counter = ADD_COUNTER_WITH_LEVEL(_profile, "BlocksProduced", TUnit::UNIT, 1);
_rows_sent_counter = ADD_COUNTER_WITH_LEVEL(_profile, "RowsProduced", TUnit::UNIT, 1);
- ((PipBufferControlBlock*)_sender.get())->set_dependency(result_sink_dependency);
+ ((PipBufferControlBlock*)_sender.get())->set_dependency(_dependency->shared_from_this());
return Status::OK();
}
diff --git a/be/src/pipeline/exec/result_sink_operator.h b/be/src/pipeline/exec/result_sink_operator.h
index 7a66738e4b4..15c9f2820b8 100644
--- a/be/src/pipeline/exec/result_sink_operator.h
+++ b/be/src/pipeline/exec/result_sink_operator.h
@@ -44,18 +44,9 @@ public:
bool can_write() override;
};
-class ResultSinkDependency final : public Dependency {
-public:
- using SharedState = BasicSharedState;
- ENABLE_FACTORY_CREATOR(ResultSinkDependency);
- ResultSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "ResultSinkDependency", true, query_ctx) {}
- ~ResultSinkDependency() override = default;
-};
-
-class ResultSinkLocalState final : public PipelineXSinkLocalState<ResultSinkDependency> {
+class ResultSinkLocalState final : public PipelineXSinkLocalState<BasicSharedState> {
ENABLE_FACTORY_CREATOR(ResultSinkLocalState);
- using Base = PipelineXSinkLocalState<ResultSinkDependency>;
+ using Base = PipelineXSinkLocalState<BasicSharedState>;
public:
ResultSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
diff --git a/be/src/pipeline/exec/scan_operator.cpp b/be/src/pipeline/exec/scan_operator.cpp
index f19bed90a9e..84e439c9032 100644
--- a/be/src/pipeline/exec/scan_operator.cpp
+++ b/be/src/pipeline/exec/scan_operator.cpp
@@ -111,14 +111,12 @@ bool ScanLocalState<Derived>::should_run_serial() const {
template <typename Derived>
Status ScanLocalState<Derived>::init(RuntimeState* state, LocalStateInfo& info) {
- RETURN_IF_ERROR(PipelineXLocalState<ScanDependency>::init(state, info));
+ RETURN_IF_ERROR(PipelineXLocalState<EmptySharedState>::init(state, info));
SCOPED_TIMER(exec_time_counter());
SCOPED_TIMER(_open_timer);
auto& p = _parent->cast<typename Derived::Parent>();
RETURN_IF_ERROR(RuntimeFilterConsumer::init(state, p.ignore_data_distribution()));
- _scan_dependency = dependency_sptr();
-
_common_expr_ctxs_push_down.resize(p._common_expr_ctxs_push_down.size());
for (size_t i = 0; i < _common_expr_ctxs_push_down.size(); i++) {
RETURN_IF_ERROR(
@@ -254,7 +252,7 @@ Status ScanLocalState<Derived>::_normalize_conjuncts() {
[&](auto&& range) {
if (range.is_empty_value_range()) {
_eos = true;
- _scan_dependency->set_ready();
+ _dependency->set_ready();
}
},
it.second.second);
@@ -546,7 +544,7 @@ template <typename Derived>
std::string ScanLocalState<Derived>::debug_string(int indentation_level) const {
fmt::memory_buffer debug_string_buffer;
fmt::format_to(debug_string_buffer, "{}, _eos = {}",
- PipelineXLocalState<ScanDependency>::debug_string(indentation_level),
+ PipelineXLocalState<EmptySharedState>::debug_string(indentation_level),
_eos.load());
if (_scanner_ctx) {
fmt::format_to(debug_string_buffer, "");
@@ -590,7 +588,7 @@ Status ScanLocalState<Derived>::_eval_const_conjuncts(vectorized::VExpr* vexpr,
if (constant_val == nullptr || !*reinterpret_cast<bool*>(constant_val)) {
*pdt = vectorized::VScanNode::PushDownType::ACCEPTABLE;
_eos = true;
- _scan_dependency->set_ready();
+ _dependency->set_ready();
}
} else if (const vectorized::ColumnVector<vectorized::UInt8>* bool_column =
check_and_get_column<vectorized::ColumnVector<vectorized::UInt8>>(
@@ -608,7 +606,7 @@ Status ScanLocalState<Derived>::_eval_const_conjuncts(vectorized::VExpr* vexpr,
if (constant_val == nullptr || !*reinterpret_cast<bool*>(constant_val)) {
*pdt = vectorized::VScanNode::PushDownType::ACCEPTABLE;
_eos = true;
- _scan_dependency->set_ready();
+ _dependency->set_ready();
}
} else {
LOG(WARNING) << "Constant predicate in scan node should return a bool column with "
@@ -806,7 +804,7 @@ Status ScanLocalState<Derived>::_normalize_not_in_and_not_eq_predicate(
auto fn_name = std::string("");
if (!is_fixed_range && state->null_in_set) {
_eos = true;
- _scan_dependency->set_ready();
+ _dependency->set_ready();
}
while (iter->has_next()) {
// column not in (nullptr) is always true
@@ -1204,7 +1202,7 @@ Status ScanLocalState<Derived>::_prepare_scanners() {
}
if (scanners.empty()) {
_eos = true;
- _scan_dependency->set_ready();
+ _dependency->set_ready();
} else {
for (auto& scanner : scanners) {
scanner->set_query_statistics(_query_statistics.get());
@@ -1221,7 +1219,7 @@ Status ScanLocalState<Derived>::_start_scanners(
auto& p = _parent->cast<typename Derived::Parent>();
_scanner_ctx = PipXScannerContext::create_shared(
state(), this, p._output_tuple_desc, p.output_row_descriptor(), scanners, p.limit(),
- state()->scan_queue_mem_limit(), _scan_dependency);
+ state()->scan_queue_mem_limit(), _dependency->shared_from_this());
if constexpr (std::is_same_v<OlapScanLocalState, Derived>) {
/**
* If `use_topn_opt` is true,
@@ -1420,7 +1418,7 @@ Status ScanLocalState<Derived>::close(RuntimeState* state) {
if (_closed) {
return Status::OK();
}
- COUNTER_UPDATE(exec_time_counter(), _scan_dependency->watcher_elapse_time());
+ COUNTER_UPDATE(exec_time_counter(), _dependency->watcher_elapse_time());
COUNTER_UPDATE(exec_time_counter(), _filter_dependency->watcher_elapse_time());
SCOPED_TIMER(_close_timer);
@@ -1428,10 +1426,10 @@ Status ScanLocalState<Derived>::close(RuntimeState* state) {
if (_scanner_ctx) {
_scanner_ctx->stop_scanners(state);
}
- COUNTER_SET(_wait_for_dependency_timer, _scan_dependency->watcher_elapse_time());
+ COUNTER_SET(_wait_for_dependency_timer, _dependency->watcher_elapse_time());
COUNTER_SET(_wait_for_rf_timer, _filter_dependency->watcher_elapse_time());
- return PipelineXLocalState<ScanDependency>::close(state);
+ return PipelineXLocalState<EmptySharedState>::close(state);
}
template <typename LocalStateType>
diff --git a/be/src/pipeline/exec/scan_operator.h b/be/src/pipeline/exec/scan_operator.h
index add2249276f..101fd3047db 100644
--- a/be/src/pipeline/exec/scan_operator.h
+++ b/be/src/pipeline/exec/scan_operator.h
@@ -57,53 +57,11 @@ public:
std::string debug_string() const override;
};
-class ScanDependency final : public Dependency {
-public:
- using SharedState = FakeSharedState;
- ENABLE_FACTORY_CREATOR(ScanDependency);
- ScanDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "ScanDependency", query_ctx) {}
-
- void block() override {
- if (_scanner_done) {
- return;
- }
- std::unique_lock<std::mutex> lc(_always_done_lock);
- if (_scanner_done) {
- return;
- }
- Dependency::block();
- }
-
- void set_scanner_done() {
- if (_scanner_done) {
- return;
- }
- std::unique_lock<std::mutex> lc(_always_done_lock);
- if (_scanner_done) {
- return;
- }
- _scanner_done = true;
- Dependency::set_ready();
- }
-
- std::string debug_string(int indentation_level = 0) override {
- fmt::memory_buffer debug_string_buffer;
- fmt::format_to(debug_string_buffer, "{}, _scanner_done = {}",
- Dependency::debug_string(indentation_level), _scanner_done);
- return fmt::to_string(debug_string_buffer);
- }
-
-private:
- bool _scanner_done {false};
- std::mutex _always_done_lock;
-};
-
-class ScanLocalStateBase : public PipelineXLocalState<ScanDependency>,
+class ScanLocalStateBase : public PipelineXLocalState<EmptySharedState>,
public vectorized::RuntimeFilterConsumer {
public:
ScanLocalStateBase(RuntimeState* state, OperatorXBase* parent)
- : PipelineXLocalState<ScanDependency>(state, parent),
+ : PipelineXLocalState<EmptySharedState>(state, parent),
vectorized::RuntimeFilterConsumer(parent->node_id(), parent->runtime_filter_descs(),
parent->row_descriptor(), _conjuncts) {}
virtual ~ScanLocalStateBase() = default;
@@ -138,7 +96,6 @@ protected:
virtual Status _init_profile() = 0;
std::atomic<bool> _opened {false};
- std::shared_ptr<ScanDependency> _scan_dependency;
std::shared_ptr<RuntimeProfile> _scanner_profile;
RuntimeProfile::Counter* _scanner_sched_counter = nullptr;
diff --git a/be/src/pipeline/exec/select_operator.h b/be/src/pipeline/exec/select_operator.h
index 84a66528875..218c8147d34 100644
--- a/be/src/pipeline/exec/select_operator.h
+++ b/be/src/pipeline/exec/select_operator.h
@@ -41,12 +41,12 @@ public:
};
class SelectOperatorX;
-class SelectLocalState final : public PipelineXLocalState<FakeDependency> {
+class SelectLocalState final : public PipelineXLocalState<FakeSharedState> {
public:
ENABLE_FACTORY_CREATOR(SelectLocalState);
SelectLocalState(RuntimeState* state, OperatorXBase* parent)
- : PipelineXLocalState<FakeDependency>(state, parent) {}
+ : PipelineXLocalState<FakeSharedState>(state, parent) {}
~SelectLocalState() = default;
private:
diff --git a/be/src/pipeline/exec/set_probe_sink_operator.cpp b/be/src/pipeline/exec/set_probe_sink_operator.cpp
index 09e5b4b0045..b746b5bf460 100644
--- a/be/src/pipeline/exec/set_probe_sink_operator.cpp
+++ b/be/src/pipeline/exec/set_probe_sink_operator.cpp
@@ -137,11 +137,12 @@ Status SetProbeSinkOperatorX<is_intersect>::sink(RuntimeState* state, vectorized
template <bool is_intersect>
Status SetProbeSinkLocalState<is_intersect>::init(RuntimeState* state, LocalSinkStateInfo& info) {
- RETURN_IF_ERROR(PipelineXSinkLocalState<SetProbeSinkDependency>::init(state, info));
+ RETURN_IF_ERROR(PipelineXSinkLocalState<SetSharedState>::init(state, info));
SCOPED_TIMER(exec_time_counter());
SCOPED_TIMER(_open_timer);
Parent& parent = _parent->cast<Parent>();
- _dependency->set_cur_child_id(parent._cur_child_id);
+ _shared_state->probe_finished_children_dependency[parent._cur_child_id] = _dependency;
+ _dependency->block();
_child_exprs.resize(parent._child_exprs.size());
for (size_t i = 0; i < _child_exprs.size(); i++) {
RETURN_IF_ERROR(parent._child_exprs[i]->clone(state, _child_exprs[i]));
diff --git a/be/src/pipeline/exec/set_probe_sink_operator.h b/be/src/pipeline/exec/set_probe_sink_operator.h
index b8687376059..a7b06267b8e 100644
--- a/be/src/pipeline/exec/set_probe_sink_operator.h
+++ b/be/src/pipeline/exec/set_probe_sink_operator.h
@@ -67,31 +67,14 @@ private:
int _child_id;
};
-class SetProbeSinkDependency final : public Dependency {
-public:
- using SharedState = SetSharedState;
- SetProbeSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "SetProbeSinkDependency", true, query_ctx) {}
- ~SetProbeSinkDependency() override = default;
-
- void set_cur_child_id(int id) {
- _child_idx = id;
- ((SetSharedState*)_shared_state)->probe_finished_children_dependency[id] = this;
- block();
- }
-
-private:
- int _child_idx {0};
-};
-
template <bool is_intersect>
class SetProbeSinkOperatorX;
template <bool is_intersect>
-class SetProbeSinkLocalState final : public PipelineXSinkLocalState<SetProbeSinkDependency> {
+class SetProbeSinkLocalState final : public PipelineXSinkLocalState<SetSharedState> {
public:
ENABLE_FACTORY_CREATOR(SetProbeSinkLocalState);
- using Base = PipelineXSinkLocalState<SetProbeSinkDependency>;
+ using Base = PipelineXSinkLocalState<SetSharedState>;
using Parent = SetProbeSinkOperatorX<is_intersect>;
SetProbeSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
diff --git a/be/src/pipeline/exec/set_sink_operator.cpp b/be/src/pipeline/exec/set_sink_operator.cpp
index 862ad411f5c..639e3335609 100644
--- a/be/src/pipeline/exec/set_sink_operator.cpp
+++ b/be/src/pipeline/exec/set_sink_operator.cpp
@@ -161,13 +161,13 @@ Status SetSinkOperatorX<is_intersect>::_extract_build_column(
template <bool is_intersect>
Status SetSinkLocalState<is_intersect>::init(RuntimeState* state, LocalSinkStateInfo& info) {
- RETURN_IF_ERROR(PipelineXSinkLocalState<SetSinkDependency>::init(state, info));
+ RETURN_IF_ERROR(PipelineXSinkLocalState<SetSharedState>::init(state, info));
SCOPED_TIMER(exec_time_counter());
SCOPED_TIMER(_open_timer);
_build_timer = ADD_TIMER(_profile, "BuildTime");
auto& parent = _parent->cast<Parent>();
- _dependency->set_cur_child_id(parent._cur_child_id);
+ _shared_state->probe_finished_children_dependency[parent._cur_child_id] = _dependency;
_child_exprs.resize(parent._child_exprs.size());
for (size_t i = 0; i < _child_exprs.size(); i++) {
RETURN_IF_ERROR(parent._child_exprs[i]->clone(state, _child_exprs[i]));
diff --git a/be/src/pipeline/exec/set_sink_operator.h b/be/src/pipeline/exec/set_sink_operator.h
index 946720cd179..a0885c18824 100644
--- a/be/src/pipeline/exec/set_sink_operator.h
+++ b/be/src/pipeline/exec/set_sink_operator.h
@@ -60,26 +60,14 @@ private:
vectorized::VSetOperationNode<is_intersect>* _set_node = nullptr;
};
-class SetSinkDependency final : public Dependency {
-public:
- using SharedState = SetSharedState;
- SetSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "SetSinkDependency", true, query_ctx) {}
- ~SetSinkDependency() override = default;
-
- void set_cur_child_id(int id) {
- ((SetSharedState*)_shared_state)->probe_finished_children_dependency[id] = this;
- }
-};
-
template <bool is_intersect>
class SetSinkOperatorX;
template <bool is_intersect>
-class SetSinkLocalState final : public PipelineXSinkLocalState<SetSinkDependency> {
+class SetSinkLocalState final : public PipelineXSinkLocalState<SetSharedState> {
public:
ENABLE_FACTORY_CREATOR(SetSinkLocalState);
- using Base = PipelineXSinkLocalState<SetSinkDependency>;
+ using Base = PipelineXSinkLocalState<SetSharedState>;
using Parent = SetSinkOperatorX<is_intersect>;
SetSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state) : Base(parent, state) {}
diff --git a/be/src/pipeline/exec/set_source_operator.cpp b/be/src/pipeline/exec/set_source_operator.cpp
index 1e64bf8a50d..db681c7e665 100644
--- a/be/src/pipeline/exec/set_source_operator.cpp
+++ b/be/src/pipeline/exec/set_source_operator.cpp
@@ -66,7 +66,7 @@ template <bool is_intersect>
Status SetSourceLocalState<is_intersect>::open(RuntimeState* state) {
SCOPED_TIMER(exec_time_counter());
SCOPED_TIMER(_open_timer);
- RETURN_IF_ERROR(PipelineXLocalState<SetSourceDependency>::open(state));
+ RETURN_IF_ERROR(PipelineXLocalState<SetSharedState>::open(state));
auto& child_exprs_lists = _shared_state->child_exprs_lists;
auto output_data_types = vectorized::VectorizedUtils::get_data_types(
diff --git a/be/src/pipeline/exec/set_source_operator.h b/be/src/pipeline/exec/set_source_operator.h
index caf3e447f56..90f09cbf97b 100644
--- a/be/src/pipeline/exec/set_source_operator.h
+++ b/be/src/pipeline/exec/set_source_operator.h
@@ -53,22 +53,14 @@ public:
Status open(RuntimeState* /*state*/) override { return Status::OK(); }
};
-class SetSourceDependency final : public Dependency {
-public:
- using SharedState = SetSharedState;
- SetSourceDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "SetSourceDependency", query_ctx) {}
- ~SetSourceDependency() override = default;
-};
-
template <bool is_intersect>
class SetSourceOperatorX;
template <bool is_intersect>
-class SetSourceLocalState final : public PipelineXLocalState<SetSourceDependency> {
+class SetSourceLocalState final : public PipelineXLocalState<SetSharedState> {
public:
ENABLE_FACTORY_CREATOR(SetSourceLocalState);
- using Base = PipelineXLocalState<SetSourceDependency>;
+ using Base = PipelineXLocalState<SetSharedState>;
using Parent = SetSourceOperatorX<is_intersect>;
SetSourceLocalState(RuntimeState* state, OperatorXBase* parent) : Base(state, parent) {};
Status init(RuntimeState* state, LocalStateInfo& infos) override;
diff --git a/be/src/pipeline/exec/sort_sink_operator.cpp b/be/src/pipeline/exec/sort_sink_operator.cpp
index 465622be4aa..4df1549990d 100644
--- a/be/src/pipeline/exec/sort_sink_operator.cpp
+++ b/be/src/pipeline/exec/sort_sink_operator.cpp
@@ -29,7 +29,7 @@ namespace doris::pipeline {
OPERATOR_CODE_GENERATOR(SortSinkOperator, StreamingOperator)
Status SortSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& info) {
- RETURN_IF_ERROR(PipelineXSinkLocalState<SortSinkDependency>::init(state, info));
+ RETURN_IF_ERROR(PipelineXSinkLocalState<SortSharedState>::init(state, info));
SCOPED_TIMER(exec_time_counter());
SCOPED_TIMER(_open_timer);
auto& p = _parent->cast<SortSinkOperatorX>();
diff --git a/be/src/pipeline/exec/sort_sink_operator.h b/be/src/pipeline/exec/sort_sink_operator.h
index d3d85a3e5c9..5502f436d36 100644
--- a/be/src/pipeline/exec/sort_sink_operator.h
+++ b/be/src/pipeline/exec/sort_sink_operator.h
@@ -45,24 +45,16 @@ public:
bool can_write() override { return true; }
};
-class SortSinkDependency final : public Dependency {
-public:
- using SharedState = SortSharedState;
- SortSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "SortSinkDependency", true, query_ctx) {}
- ~SortSinkDependency() override = default;
-};
-
enum class SortAlgorithm { HEAP_SORT, TOPN_SORT, FULL_SORT };
class SortSinkOperatorX;
-class SortSinkLocalState : public PipelineXSinkLocalState<SortSinkDependency> {
+class SortSinkLocalState : public PipelineXSinkLocalState<SortSharedState> {
ENABLE_FACTORY_CREATOR(SortSinkLocalState);
public:
SortSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
- : PipelineXSinkLocalState<SortSinkDependency>(parent, state) {}
+ : PipelineXSinkLocalState<SortSharedState>(parent, state) {}
Status init(RuntimeState* state, LocalSinkStateInfo& info) override;
diff --git a/be/src/pipeline/exec/sort_source_operator.cpp b/be/src/pipeline/exec/sort_source_operator.cpp
index 24e49d46741..2b07cb17e04 100644
--- a/be/src/pipeline/exec/sort_source_operator.cpp
+++ b/be/src/pipeline/exec/sort_source_operator.cpp
@@ -26,7 +26,7 @@ namespace doris::pipeline {
OPERATOR_CODE_GENERATOR(SortSourceOperator, SourceOperator)
SortLocalState::SortLocalState(RuntimeState* state, OperatorXBase* parent)
- : PipelineXLocalState<SortSourceDependency>(state, parent) {}
+ : PipelineXLocalState<SortSharedState>(state, parent) {}
SortSourceOperatorX::SortSourceOperatorX(ObjectPool* pool, const TPlanNode& tnode, int operator_id,
const DescriptorTbl& descs)
diff --git a/be/src/pipeline/exec/sort_source_operator.h b/be/src/pipeline/exec/sort_source_operator.h
index 216c6290700..303762b41c3 100644
--- a/be/src/pipeline/exec/sort_source_operator.h
+++ b/be/src/pipeline/exec/sort_source_operator.h
@@ -45,16 +45,8 @@ public:
Status open(RuntimeState*) override { return Status::OK(); }
};
-class SortSourceDependency final : public Dependency {
-public:
- using SharedState = SortSharedState;
- SortSourceDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "SortSourceDependency", query_ctx) {}
- ~SortSourceDependency() override = default;
-};
-
class SortSourceOperatorX;
-class SortLocalState final : public PipelineXLocalState<SortSourceDependency> {
+class SortLocalState final : public PipelineXLocalState<SortSharedState> {
public:
ENABLE_FACTORY_CREATOR(SortLocalState);
SortLocalState(RuntimeState* state, OperatorXBase* parent);
diff --git a/be/src/pipeline/exec/streaming_aggregation_operator.h b/be/src/pipeline/exec/streaming_aggregation_operator.h
index 213e63459d5..9ee8d0244a9 100644
--- a/be/src/pipeline/exec/streaming_aggregation_operator.h
+++ b/be/src/pipeline/exec/streaming_aggregation_operator.h
@@ -33,10 +33,10 @@ namespace pipeline {
class StreamingAggOperatorX;
-class StreamingAggLocalState final : public PipelineXLocalState<FakeDependency> {
+class StreamingAggLocalState final : public PipelineXLocalState<FakeSharedState> {
public:
using Parent = StreamingAggOperatorX;
- using Base = PipelineXLocalState<FakeDependency>;
+ using Base = PipelineXLocalState<FakeSharedState>;
ENABLE_FACTORY_CREATOR(StreamingAggLocalState);
StreamingAggLocalState(RuntimeState* state, OperatorXBase* parent);
~StreamingAggLocalState() override = default;
diff --git a/be/src/pipeline/exec/union_sink_operator.h b/be/src/pipeline/exec/union_sink_operator.h
index 9e7e3f8b547..fe1f26f45b2 100644
--- a/be/src/pipeline/exec/union_sink_operator.h
+++ b/be/src/pipeline/exec/union_sink_operator.h
@@ -66,23 +66,15 @@ private:
std::unique_ptr<vectorized::Block> _output_block;
};
-class UnionSinkDependency final : public Dependency {
-public:
- using SharedState = UnionSharedState;
- UnionSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "UnionSinkDependency", true, query_ctx) {}
- ~UnionSinkDependency() override = default;
-};
-
class UnionSinkOperatorX;
-class UnionSinkLocalState final : public PipelineXSinkLocalState<UnionSinkDependency> {
+class UnionSinkLocalState final : public PipelineXSinkLocalState<UnionSharedState> {
public:
ENABLE_FACTORY_CREATOR(UnionSinkLocalState);
UnionSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
: Base(parent, state), _child_row_idx(0) {}
Status init(RuntimeState* state, LocalSinkStateInfo& info) override;
friend class UnionSinkOperatorX;
- using Base = PipelineXSinkLocalState<UnionSinkDependency>;
+ using Base = PipelineXSinkLocalState<UnionSharedState>;
using Parent = UnionSinkOperatorX;
private:
diff --git a/be/src/pipeline/exec/union_source_operator.h b/be/src/pipeline/exec/union_source_operator.h
index 887c0cb9639..b58d0e442dd 100644
--- a/be/src/pipeline/exec/union_source_operator.h
+++ b/be/src/pipeline/exec/union_source_operator.h
@@ -69,19 +69,11 @@ private:
bool _need_read_for_const_expr;
};
-class UnionSourceDependency final : public Dependency {
-public:
- using SharedState = UnionSharedState;
- UnionSourceDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "UnionSourceDependency", query_ctx) {}
- ~UnionSourceDependency() override = default;
-};
-
class UnionSourceOperatorX;
-class UnionSourceLocalState final : public PipelineXLocalState<UnionSourceDependency> {
+class UnionSourceLocalState final : public PipelineXLocalState<UnionSharedState> {
public:
ENABLE_FACTORY_CREATOR(UnionSourceLocalState);
- using Base = PipelineXLocalState<UnionSourceDependency>;
+ using Base = PipelineXLocalState<UnionSharedState>;
using Parent = UnionSourceOperatorX;
UnionSourceLocalState(RuntimeState* state, OperatorXBase* parent) : Base(state, parent) {};
diff --git a/be/src/pipeline/pipeline_x/dependency.cpp b/be/src/pipeline/pipeline_x/dependency.cpp
index 9cb6e99b44f..631a17d193a 100644
--- a/be/src/pipeline/pipeline_x/dependency.cpp
+++ b/be/src/pipeline/pipeline_x/dependency.cpp
@@ -185,7 +185,7 @@ void RuntimeFilterDependency::sub_filters() {
void LocalExchangeSharedState::sub_running_sink_operators() {
std::unique_lock<std::mutex> lc(le_lock);
if (exchanger->_running_sink_operators.fetch_sub(1) == 1) {
- _set_ready_for_read();
+ _set_always_ready();
}
}
diff --git a/be/src/pipeline/pipeline_x/dependency.h b/be/src/pipeline/pipeline_x/dependency.h
index 326185c9d2b..f518ef96d46 100644
--- a/be/src/pipeline/pipeline_x/dependency.h
+++ b/be/src/pipeline/pipeline_x/dependency.h
@@ -46,8 +46,6 @@
namespace doris::pipeline {
class Dependency;
-class AnalyticSourceDependency;
-class AnalyticSinkDependency;
class PipelineXTask;
struct BasicSharedState;
using DependencySPtr = std::shared_ptr<Dependency>;
@@ -78,9 +76,8 @@ struct BasicSharedState {
};
class Dependency : public std::enable_shared_from_this<Dependency> {
- ENABLE_FACTORY_CREATOR(Dependency);
-
public:
+ ENABLE_FACTORY_CREATOR(Dependency);
Dependency(int id, int node_id, std::string name, QueryContext* query_ctx)
: _id(id),
_node_id(node_id),
@@ -137,7 +134,28 @@ public:
}
// Notify downstream pipeline tasks this dependency is blocked.
- virtual void block() { _ready = false; }
+ void block() {
+ if (_always_ready) {
+ return;
+ }
+ std::unique_lock<std::mutex> lc(_always_ready_lock);
+ if (_always_ready) {
+ return;
+ }
+ _ready = false;
+ }
+
+ void set_always_ready() {
+ if (_always_ready) {
+ return;
+ }
+ std::unique_lock<std::mutex> lc(_always_ready_lock);
+ if (_always_ready) {
+ return;
+ }
+ _always_ready = true;
+ set_ready();
+ }
protected:
void _add_block_task(PipelineXTask* task);
@@ -156,9 +174,13 @@ protected:
std::mutex _task_lock;
std::vector<PipelineXTask*> _blocked_task;
+
+ // If `_always_ready` is true, `block()` will never block tasks.
+ std::atomic<bool> _always_ready = false;
+ std::mutex _always_ready_lock;
};
-struct FakeSharedState : public BasicSharedState {};
+struct FakeSharedState final : public BasicSharedState {};
struct FakeDependency final : public Dependency {
public:
@@ -293,9 +315,13 @@ protected:
std::shared_ptr<std::atomic_bool> _blocked_by_rf;
};
+struct EmptySharedState final : public BasicSharedState {};
+
+struct AndSharedState final : public BasicSharedState {};
+
class AndDependency final : public Dependency {
public:
- using SharedState = FakeSharedState;
+ using SharedState = AndSharedState;
ENABLE_FACTORY_CREATOR(AndDependency);
AndDependency(int id, int node_id, QueryContext* query_ctx)
: Dependency(id, node_id, "AndDependency", query_ctx) {}
@@ -484,7 +510,7 @@ public:
class AsyncWriterDependency final : public Dependency {
public:
- using SharedState = FakeSharedState;
+ using SharedState = BasicSharedState;
ENABLE_FACTORY_CREATOR(AsyncWriterDependency);
AsyncWriterDependency(int id, int node_id, QueryContext* query_ctx)
: Dependency(id, node_id, "AsyncWriterDependency", true, query_ctx) {}
@@ -632,10 +658,10 @@ public:
std::atomic<size_t> mem_usage = 0;
std::mutex le_lock;
void sub_running_sink_operators();
- void _set_ready_for_read() {
+ void _set_always_ready() {
for (auto& dep : source_dependencies) {
DCHECK(dep);
- dep->set_ready();
+ dep->set_always_ready();
}
}
diff --git a/be/src/pipeline/pipeline_x/local_exchange/local_exchange_sink_operator.h b/be/src/pipeline/pipeline_x/local_exchange/local_exchange_sink_operator.h
index 7ed7d7d0e90..2c7fe73c871 100644
--- a/be/src/pipeline/pipeline_x/local_exchange/local_exchange_sink_operator.h
+++ b/be/src/pipeline/pipeline_x/local_exchange/local_exchange_sink_operator.h
@@ -22,24 +22,15 @@
namespace doris::pipeline {
-struct LocalExchangeSinkDependency final : public Dependency {
-public:
- using SharedState = LocalExchangeSharedState;
- LocalExchangeSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "LocalExchangeSinkDependency", true, query_ctx) {}
- ~LocalExchangeSinkDependency() override = default;
-};
-
class Exchanger;
class ShuffleExchanger;
class PassthroughExchanger;
class BroadcastExchanger;
class PassToOneExchanger;
class LocalExchangeSinkOperatorX;
-class LocalExchangeSinkLocalState final
- : public PipelineXSinkLocalState<LocalExchangeSinkDependency> {
+class LocalExchangeSinkLocalState final : public PipelineXSinkLocalState<LocalExchangeSharedState> {
public:
- using Base = PipelineXSinkLocalState<LocalExchangeSinkDependency>;
+ using Base = PipelineXSinkLocalState<LocalExchangeSharedState>;
ENABLE_FACTORY_CREATOR(LocalExchangeSinkLocalState);
LocalExchangeSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
diff --git a/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.cpp b/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.cpp
index edfb8114811..b53fead5c5a 100644
--- a/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.cpp
+++ b/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.cpp
@@ -21,17 +21,6 @@
namespace doris::pipeline {
-void LocalExchangeSourceDependency::block() {
- if (((LocalExchangeSharedState*)_shared_state)->exchanger->_running_sink_operators == 0) {
- return;
- }
- std::unique_lock<std::mutex> lc(((LocalExchangeSharedState*)_shared_state)->le_lock);
- if (((LocalExchangeSharedState*)_shared_state)->exchanger->_running_sink_operators == 0) {
- return;
- }
- Dependency::block();
-}
-
Status LocalExchangeSourceLocalState::init(RuntimeState* state, LocalStateInfo& info) {
RETURN_IF_ERROR(Base::init(state, info));
SCOPED_TIMER(exec_time_counter());
diff --git a/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.h b/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.h
index b62ec1a8fb9..1a9614de352 100644
--- a/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.h
+++ b/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.h
@@ -22,26 +22,15 @@
namespace doris::pipeline {
-struct LocalExchangeSourceDependency final : public Dependency {
-public:
- using SharedState = LocalExchangeSharedState;
- LocalExchangeSourceDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "LocalExchangeSourceDependency", query_ctx) {}
- ~LocalExchangeSourceDependency() override = default;
-
- void block() override;
-};
-
class Exchanger;
class ShuffleExchanger;
class PassthroughExchanger;
class BroadcastExchanger;
class PassToOneExchanger;
class LocalExchangeSourceOperatorX;
-class LocalExchangeSourceLocalState final
- : public PipelineXLocalState<LocalExchangeSourceDependency> {
+class LocalExchangeSourceLocalState final : public PipelineXLocalState<LocalExchangeSharedState> {
public:
- using Base = PipelineXLocalState<LocalExchangeSourceDependency>;
+ using Base = PipelineXLocalState<LocalExchangeSharedState>;
ENABLE_FACTORY_CREATOR(LocalExchangeSourceLocalState);
LocalExchangeSourceLocalState(RuntimeState* state, OperatorXBase* parent)
: Base(state, parent) {}
diff --git a/be/src/pipeline/pipeline_x/local_exchange/local_exchanger.h b/be/src/pipeline/pipeline_x/local_exchange/local_exchanger.h
index 28c99e751ea..a27a592f8c5 100644
--- a/be/src/pipeline/pipeline_x/local_exchange/local_exchanger.h
+++ b/be/src/pipeline/pipeline_x/local_exchange/local_exchanger.h
@@ -47,7 +47,6 @@ public:
virtual ExchangeType get_type() const = 0;
protected:
- friend struct LocalExchangeSourceDependency;
friend struct LocalExchangeSharedState;
friend struct ShuffleBlockWrapper;
friend class LocalExchangeSourceLocalState;
diff --git a/be/src/pipeline/pipeline_x/operator.cpp b/be/src/pipeline/pipeline_x/operator.cpp
index 890c4298bb0..101b02cf406 100644
--- a/be/src/pipeline/pipeline_x/operator.cpp
+++ b/be/src/pipeline/pipeline_x/operator.cpp
@@ -71,15 +71,15 @@
namespace doris::pipeline {
-template <typename DependencyType>
-std::string PipelineXLocalState<DependencyType>::debug_string(int indentation_level) const {
+template <typename SharedStateArg>
+std::string PipelineXLocalState<SharedStateArg>::debug_string(int indentation_level) const {
fmt::memory_buffer debug_string_buffer;
fmt::format_to(debug_string_buffer, "{}", _parent->debug_string(indentation_level));
return fmt::to_string(debug_string_buffer);
}
-template <typename DependencyType>
-std::string PipelineXSinkLocalState<DependencyType>::debug_string(int indentation_level) const {
+template <typename SharedStateArg>
+std::string PipelineXSinkLocalState<SharedStateArg>::debug_string(int indentation_level) const {
fmt::memory_buffer debug_string_buffer;
fmt::format_to(debug_string_buffer, "{}", _parent->debug_string(indentation_level));
return fmt::to_string(debug_string_buffer);
@@ -283,14 +283,13 @@ void DataSinkOperatorX<LocalStateType>::get_dependency(
vector<DependencySPtr>& dependency,
std::map<int, std::shared_ptr<BasicSharedState>>& shared_states, QueryContext* ctx) {
std::shared_ptr<BasicSharedState> ss = nullptr;
- if constexpr (NeedToCreate<typename LocalStateType::DependencyType::SharedState>) {
- ss.reset(new typename LocalStateType::DependencyType::SharedState());
+ if constexpr (NeedToCreate<typename LocalStateType::SharedStateType>) {
+ ss.reset(new typename LocalStateType::SharedStateType());
DCHECK(!shared_states.contains(dests_id().front()));
- if constexpr (!std::is_same_v<typename LocalStateType::DependencyType::SharedState,
- FakeSharedState>) {
+ if constexpr (!std::is_same_v<typename LocalStateType::SharedStateType, FakeSharedState>) {
shared_states.insert({dests_id().front(), ss});
}
- } else if constexpr (std::is_same_v<typename LocalStateType::DependencyType::SharedState,
+ } else if constexpr (std::is_same_v<typename LocalStateType::SharedStateType,
MultiCastSharedState>) {
ss = ((MultiCastDataStreamSinkOperatorX*)this)->create_multi_cast_data_streamer();
auto& dests = dests_id();
@@ -299,11 +298,18 @@ void DataSinkOperatorX<LocalStateType>::get_dependency(
shared_states.insert({dest_id, ss});
}
}
- if constexpr (!std::is_same_v<typename LocalStateType::DependencyType, FakeDependency>) {
+ if constexpr (std::is_same_v<typename LocalStateType::SharedStateType, AndSharedState>) {
auto& dests = dests_id();
for (auto& dest_id : dests) {
- dependency.push_back(std::make_shared<typename LocalStateType::DependencyType>(
- dest_id, _node_id, ctx));
+ dependency.push_back(std::make_shared<AndDependency>(dest_id, _node_id, ctx));
+ dependency.back()->set_shared_state(ss.get());
+ }
+ } else if constexpr (!std::is_same_v<typename LocalStateType::SharedStateType,
+ FakeSharedState>) {
+ auto& dests = dests_id();
+ for (auto& dest_id : dests) {
+ dependency.push_back(std::make_shared<Dependency>(dest_id, _node_id,
+ _name + "_DEPENDENCY", true, ctx));
dependency.back()->set_shared_state(ss.get());
}
} else {
@@ -315,19 +321,25 @@ template <typename LocalStateType>
DependencySPtr OperatorX<LocalStateType>::get_dependency(
QueryContext* ctx, std::map<int, std::shared_ptr<BasicSharedState>>& shared_states) {
std::shared_ptr<BasicSharedState> ss = nullptr;
- if constexpr (std::is_same_v<typename LocalStateType::DependencyType::SharedState,
- SetSharedState>) {
- ss.reset(new typename LocalStateType::DependencyType::SharedState());
+ if constexpr (std::is_same_v<typename LocalStateType::SharedStateType, SetSharedState>) {
+ ss.reset(new typename LocalStateType::SharedStateType());
shared_states.insert({operator_id(), ss});
- } else if constexpr (std::is_same_v<typename LocalStateType::DependencyType::SharedState,
+ } else if constexpr (std::is_same_v<typename LocalStateType::SharedStateType,
UnionSharedState>) {
- ss.reset(new typename LocalStateType::DependencyType::SharedState(
+ ss.reset(new typename LocalStateType::SharedStateType(
((UnionSourceOperatorX*)this)->get_child_count()));
shared_states.insert({operator_id(), ss});
}
- auto dep =
- std::make_shared<typename LocalStateType::DependencyType>(_operator_id, _node_id, ctx);
- dep->set_shared_state(ss.get());
+ DependencySPtr dep = nullptr;
+ if constexpr (std::is_same_v<typename LocalStateType::SharedStateType, AndSharedState>) {
+ dep = std::make_shared<AndDependency>(_operator_id, _node_id, ctx);
+ } else if constexpr (std::is_same_v<typename LocalStateType::SharedStateType,
+ FakeSharedState>) {
+ dep = std::make_shared<FakeDependency>(_operator_id, _node_id, ctx);
+ } else {
+ dep = std::make_shared<Dependency>(_operator_id, _node_id, _op_name + "_DEPENDENCY", ctx);
+ dep->set_shared_state(ss.get());
+ }
return dep;
}
@@ -354,29 +366,27 @@ PipelineXLocalStateBase::PipelineXLocalStateBase(RuntimeState* state, OperatorXB
_query_statistics = std::make_shared<QueryStatistics>();
}
-template <typename DependencyType>
-Status PipelineXLocalState<DependencyType>::init(RuntimeState* state, LocalStateInfo& info) {
+template <typename SharedStateArg>
+Status PipelineXLocalState<SharedStateArg>::init(RuntimeState* state, LocalStateInfo& info) {
_runtime_profile.reset(new RuntimeProfile(_parent->get_name() + name_suffix()));
_runtime_profile->set_metadata(_parent->node_id());
_runtime_profile->set_is_sink(false);
info.parent_profile->add_child(_runtime_profile.get(), true, nullptr);
- constexpr auto is_fake_shared =
- std::is_same_v<typename DependencyType::SharedState, FakeSharedState>;
- _dependency = (DependencyType*)info.dependency.get();
- if constexpr (!std::is_same_v<FakeDependency, DependencyType>) {
+ constexpr auto is_fake_shared = std::is_same_v<SharedStateArg, FakeSharedState>;
+ _dependency = info.dependency.get();
+ if constexpr (!is_fake_shared) {
_wait_for_dependency_timer = ADD_TIMER_WITH_LEVEL(
_runtime_profile, "WaitForDependency[" + _dependency->name() + "]Time", 1);
auto& deps = info.upstream_dependencies;
- if constexpr (std::is_same_v<LocalExchangeSourceDependency, DependencyType>) {
+ if constexpr (std::is_same_v<LocalExchangeSharedState, SharedStateArg>) {
_dependency->set_shared_state(info.le_state_map[_parent->operator_id()].first.get());
- _shared_state = _dependency->shared_state()
- ->template cast<typename DependencyType::SharedState>();
+ _shared_state = _dependency->shared_state()->template cast<SharedStateArg>();
_shared_state->source_dep = info.dependency;
- } else if constexpr (!is_fake_shared) {
+ } else if constexpr (!std::is_same_v<SharedStateArg, EmptySharedState> &&
+ !std::is_same_v<SharedStateArg, AndSharedState>) {
_dependency->set_shared_state(info.shared_state);
- _shared_state = _dependency->shared_state()
- ->template cast<typename DependencyType::SharedState>();
+ _shared_state = _dependency->shared_state()->template cast<SharedStateArg>();
_shared_state->source_dep = info.dependency;
if (!deps.empty()) {
@@ -408,12 +418,12 @@ Status PipelineXLocalState<DependencyType>::init(RuntimeState* state, LocalState
return Status::OK();
}
-template <typename DependencyType>
-Status PipelineXLocalState<DependencyType>::close(RuntimeState* state) {
+template <typename SharedStateArg>
+Status PipelineXLocalState<SharedStateArg>::close(RuntimeState* state) {
if (_closed) {
return Status::OK();
}
- if constexpr (!std::is_same_v<DependencyType, FakeDependency>) {
+ if constexpr (!std::is_same_v<SharedStateArg, FakeSharedState>) {
COUNTER_SET(_wait_for_dependency_timer, _dependency->watcher_elapse_time());
}
if (_rows_returned_counter != nullptr) {
@@ -426,34 +436,29 @@ Status PipelineXLocalState<DependencyType>::close(RuntimeState* state) {
return Status::OK();
}
-template <typename DependencyType>
-Status PipelineXSinkLocalState<DependencyType>::init(RuntimeState* state,
- LocalSinkStateInfo& info) {
+template <typename SharedState>
+Status PipelineXSinkLocalState<SharedState>::init(RuntimeState* state, LocalSinkStateInfo& info) {
// create profile
_profile = state->obj_pool()->add(new RuntimeProfile(_parent->get_name() + name_suffix()));
_profile->set_metadata(_parent->node_id());
_profile->set_is_sink(true);
_wait_for_finish_dependency_timer = ADD_TIMER(_profile, "PendingFinishDependency");
- constexpr auto is_fake_shared =
- std::is_same_v<typename DependencyType::SharedState, FakeSharedState>;
- if constexpr (!std::is_same_v<FakeDependency, DependencyType>) {
+ constexpr auto is_fake_shared = std::is_same_v<SharedState, FakeSharedState>;
+ if constexpr (!is_fake_shared) {
auto& deps = info.dependencies;
- _dependency = (DependencyType*)deps.front().get();
- if constexpr (std::is_same_v<LocalExchangeSinkDependency, DependencyType>) {
+ _dependency = deps.front().get();
+ if constexpr (std::is_same_v<LocalExchangeSharedState, SharedState>) {
_dependency = info.le_state_map[_parent->dests_id().front()].second.get();
}
if (_dependency) {
- if constexpr (!is_fake_shared) {
- _shared_state = (typename DependencyType::SharedState*)_dependency->shared_state();
- }
-
+ _shared_state = (SharedState*)_dependency->shared_state();
_wait_for_dependency_timer = ADD_TIMER_WITH_LEVEL(
_profile, "WaitForDependency[" + _dependency->name() + "]Time", 1);
}
} else {
auto& deps = info.dependencies;
deps.front() = std::make_shared<FakeDependency>(0, 0, state->get_query_ctx());
- _dependency = (DependencyType*)deps.front().get();
+ _dependency = deps.front().get();
}
_rows_input_counter = ADD_COUNTER_WITH_LEVEL(_profile, "InputRows", TUnit::UNIT, 1);
_open_timer = ADD_TIMER_WITH_LEVEL(_profile, "OpenTime", 1);
@@ -467,12 +472,12 @@ Status PipelineXSinkLocalState<DependencyType>::init(RuntimeState* state,
return Status::OK();
}
-template <typename DependencyType>
-Status PipelineXSinkLocalState<DependencyType>::close(RuntimeState* state, Status exec_status) {
+template <typename SharedState>
+Status PipelineXSinkLocalState<SharedState>::close(RuntimeState* state, Status exec_status) {
if (_closed) {
return Status::OK();
}
- if constexpr (!std::is_same_v<DependencyType, FakeDependency>) {
+ if constexpr (!std::is_same_v<SharedState, FakeSharedState>) {
COUNTER_SET(_wait_for_dependency_timer, _dependency->watcher_elapse_time());
}
if (_peak_memory_usage_counter) {
@@ -646,34 +651,34 @@ template class StatefulOperatorX<DistinctStreamingAggLocalState>;
template class StatefulOperatorX<NestedLoopJoinProbeLocalState>;
template class StatefulOperatorX<TableFunctionLocalState>;
-template class PipelineXSinkLocalState<SharedHashTableDependency>;
-template class PipelineXSinkLocalState<SortSinkDependency>;
-template class PipelineXSinkLocalState<NestedLoopJoinBuildSinkDependency>;
-template class PipelineXSinkLocalState<AnalyticSinkDependency>;
-template class PipelineXSinkLocalState<AggSinkDependency>;
-template class PipelineXSinkLocalState<FakeDependency>;
-template class PipelineXSinkLocalState<UnionSinkDependency>;
-template class PipelineXSinkLocalState<PartitionSortSinkDependency>;
-template class PipelineXSinkLocalState<MultiCastSinkDependency>;
-template class PipelineXSinkLocalState<SetSinkDependency>;
-template class PipelineXSinkLocalState<SetProbeSinkDependency>;
-template class PipelineXSinkLocalState<LocalExchangeSinkDependency>;
-template class PipelineXSinkLocalState<AndDependency>;
-template class PipelineXSinkLocalState<ResultSinkDependency>;
-
-template class PipelineXLocalState<HashJoinProbeDependency>;
-template class PipelineXLocalState<SortSourceDependency>;
-template class PipelineXLocalState<NestedLoopJoinProbeDependency>;
-template class PipelineXLocalState<AnalyticSourceDependency>;
-template class PipelineXLocalState<AggSourceDependency>;
-template class PipelineXLocalState<FakeDependency>;
-template class PipelineXLocalState<UnionSourceDependency>;
-template class PipelineXLocalState<MultiCastSourceDependency>;
-template class PipelineXLocalState<PartitionSortSourceDependency>;
-template class PipelineXLocalState<SetSourceDependency>;
-template class PipelineXLocalState<LocalExchangeSourceDependency>;
-template class PipelineXLocalState<AndDependency>;
-template class PipelineXLocalState<ScanDependency>;
+template class PipelineXSinkLocalState<HashJoinSharedState>;
+template class PipelineXSinkLocalState<SortSharedState>;
+template class PipelineXSinkLocalState<NestedLoopJoinSharedState>;
+template class PipelineXSinkLocalState<AnalyticSharedState>;
+template class PipelineXSinkLocalState<AggSharedState>;
+template class PipelineXSinkLocalState<FakeSharedState>;
+template class PipelineXSinkLocalState<UnionSharedState>;
+template class PipelineXSinkLocalState<PartitionSortNodeSharedState>;
+template class PipelineXSinkLocalState<MultiCastSharedState>;
+template class PipelineXSinkLocalState<SetSharedState>;
+template class PipelineXSinkLocalState<AndSharedState>;
+template class PipelineXSinkLocalState<LocalExchangeSharedState>;
+template class PipelineXSinkLocalState<BasicSharedState>;
+
+template class PipelineXLocalState<HashJoinSharedState>;
+template class PipelineXLocalState<SortSharedState>;
+template class PipelineXLocalState<NestedLoopJoinSharedState>;
+template class PipelineXLocalState<AnalyticSharedState>;
+template class PipelineXLocalState<AggSharedState>;
+template class PipelineXLocalState<FakeSharedState>;
+template class PipelineXLocalState<UnionSharedState>;
+template class PipelineXLocalState<MultiCastSharedState>;
+template class PipelineXLocalState<PartitionSortNodeSharedState>;
+template class PipelineXLocalState<SetSharedState>;
+template class PipelineXLocalState<LocalExchangeSharedState>;
+template class PipelineXLocalState<EmptySharedState>;
+template class PipelineXLocalState<AndSharedState>;
+template class PipelineXLocalState<BasicSharedState>;
template class AsyncWriterSink<doris::vectorized::VFileResultWriter, ResultFileSinkOperatorX>;
template class AsyncWriterSink<doris::vectorized::VJdbcTableWriter, JdbcTableSinkOperatorX>;
diff --git a/be/src/pipeline/pipeline_x/operator.h b/be/src/pipeline/pipeline_x/operator.h
index 848ed5cd001..87101de0612 100644
--- a/be/src/pipeline/pipeline_x/operator.h
+++ b/be/src/pipeline/pipeline_x/operator.h
@@ -27,16 +27,13 @@ class AsyncResultWriter;
}
namespace doris::pipeline {
-struct LocalExchangeSinkDependency;
-
// This struct is used only for initializing local state.
struct LocalStateInfo {
RuntimeProfile* parent_profile = nullptr;
const std::vector<TScanRangeParams> scan_ranges;
std::vector<DependencySPtr>& upstream_dependencies;
BasicSharedState* shared_state;
- std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>,
- std::shared_ptr<LocalExchangeSinkDependency>>>
+ std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>, std::shared_ptr<Dependency>>>
le_state_map;
const int task_idx;
@@ -49,8 +46,7 @@ struct LocalSinkStateInfo {
RuntimeProfile* parent_profile = nullptr;
const int sender_id;
std::vector<DependencySPtr>& dependencies;
- std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>,
- std::shared_ptr<LocalExchangeSinkDependency>>>
+ std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>, std::shared_ptr<Dependency>>>
le_state_map;
const TDataSink& tsink;
};
@@ -347,10 +343,10 @@ public:
std::map<int, std::shared_ptr<BasicSharedState>>& shared_states) override;
};
-template <typename DependencyArg = FakeDependency>
+template <typename SharedStateArg = FakeSharedState>
class PipelineXLocalState : public PipelineXLocalStateBase {
public:
- using DependencyType = DependencyArg;
+ using SharedStateType = SharedStateArg;
PipelineXLocalState(RuntimeState* state, OperatorXBase* parent)
: PipelineXLocalStateBase(state, parent) {}
~PipelineXLocalState() override = default;
@@ -367,13 +363,9 @@ public:
Dependency* dependency() override { return _dependency; }
- auto dependency_sptr() {
- return std::dynamic_pointer_cast<DependencyArg>(_dependency->shared_from_this());
- }
-
protected:
- DependencyType* _dependency = nullptr;
- typename DependencyType::SharedState* _shared_state = nullptr;
+ Dependency* _dependency = nullptr;
+ SharedStateArg* _shared_state = nullptr;
};
class DataSinkOperatorXBase;
@@ -612,10 +604,10 @@ public:
}
};
-template <typename DependencyArg = FakeDependency>
+template <typename SharedStateArg = FakeSharedState>
class PipelineXSinkLocalState : public PipelineXSinkLocalStateBase {
public:
- using DependencyType = DependencyArg;
+ using SharedStateType = SharedStateArg;
PipelineXSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
: PipelineXSinkLocalStateBase(parent, state) {}
~PipelineXSinkLocalState() override = default;
@@ -632,13 +624,9 @@ public:
Dependency* dependency() override { return _dependency; }
- auto dependency_sptr() {
- return std::dynamic_pointer_cast<DependencyArg>(_dependency->shared_from_this());
- }
-
protected:
- DependencyType* _dependency = nullptr;
- typename DependencyType::SharedState* _shared_state = nullptr;
+ Dependency* _dependency = nullptr;
+ SharedStateType* _shared_state = nullptr;
};
/**
@@ -687,9 +675,9 @@ public:
template <typename Writer, typename Parent>
requires(std::is_base_of_v<vectorized::AsyncResultWriter, Writer>)
-class AsyncWriterSink : public PipelineXSinkLocalState<FakeDependency> {
+class AsyncWriterSink : public PipelineXSinkLocalState<FakeSharedState> {
public:
- using Base = PipelineXSinkLocalState<FakeDependency>;
+ using Base = PipelineXSinkLocalState<FakeSharedState>;
AsyncWriterSink(DataSinkOperatorXBase* parent, RuntimeState* state)
: Base(parent, state), _async_writer_dependency(nullptr) {
_finish_dependency = std::make_shared<FinishDependency>(
diff --git a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
index 693cb4d5c30..efca7c068b6 100644
--- a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -537,9 +537,9 @@ Status PipelineXFragmentContext::_build_pipeline_tasks(
std::map<PipelineId, PipelineXTask*> pipeline_id_to_task;
auto get_local_exchange_state = [&](PipelinePtr pipeline)
-> std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>,
- std::shared_ptr<LocalExchangeSinkDependency>>> {
+ std::shared_ptr<Dependency>>> {
std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>,
- std::shared_ptr<LocalExchangeSinkDependency>>>
+ std::shared_ptr<Dependency>>>
le_state_map;
auto source_id = pipeline->operator_xs().front()->operator_id();
if (auto iter = _op_id_to_le_state.find(source_id); iter != _op_id_to_le_state.end()) {
@@ -774,8 +774,9 @@ Status PipelineXFragmentContext::_add_local_exchange_impl(
return Status::InternalError("Unsupported local exchange type : " +
std::to_string((int)data_distribution.distribution_type));
}
- auto sink_dep = std::make_shared<LocalExchangeSinkDependency>(sink_id, local_exchange_id,
- _runtime_state->get_query_ctx());
+ auto sink_dep = std::make_shared<Dependency>(sink_id, local_exchange_id,
+ "LOCAL_EXCHANGE_SINK_DEPENDENCY", true,
+ _runtime_state->get_query_ctx());
sink_dep->set_shared_state(shared_state.get());
shared_state->sink_dependency = sink_dep;
_op_id_to_le_state.insert({local_exchange_id, {shared_state, sink_dep}});
diff --git a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.h b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.h
index 54714dd4665..102dd854998 100644
--- a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.h
+++ b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.h
@@ -53,7 +53,7 @@ class TDataSink;
class TPipelineFragmentParams;
namespace pipeline {
-struct LocalExchangeSinkDependency;
+class Dependency;
class PipelineXFragmentContext : public PipelineFragmentContext {
public:
@@ -227,8 +227,7 @@ private:
int _operator_id = 0;
int _sink_operator_id = 0;
- std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>,
- std::shared_ptr<LocalExchangeSinkDependency>>>
+ std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>, std::shared_ptr<Dependency>>>
_op_id_to_le_state;
// UniqueId -> runtime mgr
diff --git a/be/src/pipeline/pipeline_x/pipeline_x_task.cpp b/be/src/pipeline/pipeline_x/pipeline_x_task.cpp
index 529eff5068f..7579789534f 100644
--- a/be/src/pipeline/pipeline_x/pipeline_x_task.cpp
+++ b/be/src/pipeline/pipeline_x/pipeline_x_task.cpp
@@ -44,13 +44,13 @@ class RuntimeState;
namespace doris::pipeline {
-PipelineXTask::PipelineXTask(PipelinePtr& pipeline, uint32_t task_id, RuntimeState* state,
- PipelineFragmentContext* fragment_context,
- RuntimeProfile* parent_profile,
- std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>,
- std::shared_ptr<LocalExchangeSinkDependency>>>
- le_state_map,
- int task_idx)
+PipelineXTask::PipelineXTask(
+ PipelinePtr& pipeline, uint32_t task_id, RuntimeState* state,
+ PipelineFragmentContext* fragment_context, RuntimeProfile* parent_profile,
+ std::map<int,
+ std::pair<std::shared_ptr<LocalExchangeSharedState>, std::shared_ptr<Dependency>>>
+ le_state_map,
+ int task_idx)
: PipelineTask(pipeline, task_id, state, fragment_context, parent_profile),
_operators(pipeline->operator_xs()),
_source(_operators.front()),
diff --git a/be/src/pipeline/pipeline_x/pipeline_x_task.h b/be/src/pipeline/pipeline_x/pipeline_x_task.h
index a558dbeb40e..6e8a3773751 100644
--- a/be/src/pipeline/pipeline_x/pipeline_x_task.h
+++ b/be/src/pipeline/pipeline_x/pipeline_x_task.h
@@ -46,7 +46,7 @@ namespace doris::pipeline {
class TaskQueue;
class PriorityTaskQueue;
-struct LocalExchangeSinkDependency;
+class Dependency;
// The class do the pipeline task. Minest schdule union by task scheduler
class PipelineXTask : public PipelineTask {
@@ -54,7 +54,7 @@ public:
PipelineXTask(PipelinePtr& pipeline, uint32_t task_id, RuntimeState* state,
PipelineFragmentContext* fragment_context, RuntimeProfile* parent_profile,
std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>,
- std::shared_ptr<LocalExchangeSinkDependency>>>
+ std::shared_ptr<Dependency>>>
le_state_map,
int task_idx);
@@ -215,8 +215,7 @@ private:
std::vector<DependencySPtr> _downstream_dependency;
// All shared states of this pipeline task.
std::map<int, std::shared_ptr<BasicSharedState>> _shared_states;
- std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>,
- std::shared_ptr<LocalExchangeSinkDependency>>>
+ std::map<int, std::pair<std::shared_ptr<LocalExchangeSharedState>, std::shared_ptr<Dependency>>>
_le_state_map;
int _task_idx;
bool _dry_run = false;
diff --git a/be/src/runtime/buffer_control_block.cpp b/be/src/runtime/buffer_control_block.cpp
index d746a6aca10..c2ec1831c7e 100644
--- a/be/src/runtime/buffer_control_block.cpp
+++ b/be/src/runtime/buffer_control_block.cpp
@@ -300,7 +300,7 @@ Status PipBufferControlBlock::cancel() {
}
void PipBufferControlBlock::set_dependency(
- std::shared_ptr<pipeline::ResultSinkDependency> result_sink_dependency) {
+ std::shared_ptr<pipeline::Dependency> result_sink_dependency) {
_result_sink_dependency = result_sink_dependency;
}
diff --git a/be/src/runtime/buffer_control_block.h b/be/src/runtime/buffer_control_block.h
index d84bcba6818..b691b4640dd 100644
--- a/be/src/runtime/buffer_control_block.h
+++ b/be/src/runtime/buffer_control_block.h
@@ -48,7 +48,7 @@ class Controller;
namespace doris {
namespace pipeline {
-class ResultSinkDependency;
+class Dependency;
} // namespace pipeline
class PFetchDataResult;
@@ -154,7 +154,7 @@ public:
Status cancel() override;
- void set_dependency(std::shared_ptr<pipeline::ResultSinkDependency> result_sink_dependency);
+ void set_dependency(std::shared_ptr<pipeline::Dependency> result_sink_dependency);
private:
void _update_dependency();
@@ -162,7 +162,7 @@ private:
void _update_batch_queue_empty() override;
std::atomic_bool _batch_queue_empty {false};
- std::shared_ptr<pipeline::ResultSinkDependency> _result_sink_dependency;
+ std::shared_ptr<pipeline::Dependency> _result_sink_dependency;
};
} // namespace doris
diff --git a/be/src/vec/exec/scan/pip_scanner_context.h b/be/src/vec/exec/scan/pip_scanner_context.h
index b69f7c031d4..e33523cf3bf 100644
--- a/be/src/vec/exec/scan/pip_scanner_context.h
+++ b/be/src/vec/exec/scan/pip_scanner_context.h
@@ -47,7 +47,7 @@ public:
const RowDescriptor* output_row_descriptor,
const std::list<std::shared_ptr<vectorized::ScannerDelegate>>& scanners,
int64_t limit_, int64_t max_bytes_in_blocks_queue,
- std::shared_ptr<pipeline::ScanDependency> dependency)
+ std::shared_ptr<pipeline::Dependency> dependency)
: vectorized::ScannerContext(state, output_tuple_desc, output_row_descriptor, scanners,
limit_, max_bytes_in_blocks_queue, 1, local_state) {
_dependency = dependency;
@@ -75,12 +75,12 @@ public:
protected:
void _set_scanner_done() override {
if (_dependency) {
- _dependency->set_scanner_done();
+ _dependency->set_always_ready();
}
}
private:
- std::shared_ptr<pipeline::ScanDependency> _dependency = nullptr;
+ std::shared_ptr<pipeline::Dependency> _dependency = nullptr;
};
} // namespace doris::pipeline
diff --git a/be/src/vec/exec/scan/scanner_context.h b/be/src/vec/exec/scan/scanner_context.h
index 58e6f4dae7f..ca80d6e748c 100644
--- a/be/src/vec/exec/scan/scanner_context.h
+++ b/be/src/vec/exec/scan/scanner_context.h
@@ -42,7 +42,6 @@ class TupleDescriptor;
namespace pipeline {
class ScanLocalStateBase;
-class ScanDependency;
class Dependency;
} // namespace pipeline
diff --git a/be/src/vec/runtime/shared_hash_table_controller.h b/be/src/vec/runtime/shared_hash_table_controller.h
index a81fc994f39..1f48ffa3439 100644
--- a/be/src/vec/runtime/shared_hash_table_controller.h
+++ b/be/src/vec/runtime/shared_hash_table_controller.h
@@ -37,7 +37,7 @@ class BloomFilterFuncBase;
class BitmapFilterFuncBase;
namespace pipeline {
-class SharedHashTableDependency;
+class Dependency;
}
namespace vectorized {
@@ -78,7 +78,7 @@ public:
Status wait_for_signal(RuntimeState* state, const SharedHashTableContextPtr& context);
bool should_build_hash_table(const TUniqueId& fragment_instance_id, int my_node_id);
void set_pipeline_engine_enabled(bool enabled) { _pipeline_engine_enabled = enabled; }
- void append_dependency(int node_id, std::shared_ptr<pipeline::SharedHashTableDependency> dep) {
+ void append_dependency(int node_id, std::shared_ptr<pipeline::Dependency> dep) {
std::lock_guard<std::mutex> lock(_mutex);
_dependencies[node_id].push_back(dep);
}
@@ -87,8 +87,7 @@ private:
bool _pipeline_engine_enabled = false;
std::mutex _mutex;
// For pipelineX, we update all dependencies once hash table is built;
- std::map<int /*node id*/, std::vector<std::shared_ptr<pipeline::SharedHashTableDependency>>>
- _dependencies;
+ std::map<int /*node id*/, std::vector<std::shared_ptr<pipeline::Dependency>>> _dependencies;
std::condition_variable _cv;
std::map<int /*node id*/, TUniqueId /*fragment instance id*/> _builder_fragment_ids;
std::map<int /*node id*/, SharedHashTableContextPtr> _shared_contexts;
diff --git a/be/src/vec/runtime/vdata_stream_recvr.cpp b/be/src/vec/runtime/vdata_stream_recvr.cpp
index 6c26128bf5a..7ee7f419fff 100644
--- a/be/src/vec/runtime/vdata_stream_recvr.cpp
+++ b/be/src/vec/runtime/vdata_stream_recvr.cpp
@@ -360,9 +360,9 @@ VDataStreamRecvr::VDataStreamRecvr(VDataStreamMgr* stream_mgr, RuntimeState* sta
if (state->enable_pipeline_x_exec()) {
_sender_to_local_channel_dependency.resize(num_queues);
for (size_t i = 0; i < num_queues; i++) {
- _sender_to_local_channel_dependency[i] =
- pipeline::LocalExchangeChannelDependency::create_shared(
- _dest_node_id, _dest_node_id, state->get_query_ctx());
+ _sender_to_local_channel_dependency[i] = pipeline::Dependency::create_shared(
+ _dest_node_id, _dest_node_id, "LocalExchangeChannelDependency", true,
+ state->get_query_ctx());
}
}
_sender_queues.reserve(num_queues);
@@ -441,8 +441,8 @@ bool VDataStreamRecvr::sender_queue_empty(int sender_id) {
return _sender_queues[use_sender_id]->queue_empty();
}
-std::shared_ptr<pipeline::LocalExchangeChannelDependency>
-VDataStreamRecvr::get_local_channel_dependency(int sender_id) {
+std::shared_ptr<pipeline::Dependency> VDataStreamRecvr::get_local_channel_dependency(
+ int sender_id) {
DCHECK(_sender_to_local_channel_dependency[_is_merging ? sender_id : 0] != nullptr);
return _sender_to_local_channel_dependency[_is_merging ? sender_id : 0];
}
diff --git a/be/src/vec/runtime/vdata_stream_recvr.h b/be/src/vec/runtime/vdata_stream_recvr.h
index 30ac4088f88..9ce8d72d73a 100644
--- a/be/src/vec/runtime/vdata_stream_recvr.h
+++ b/be/src/vec/runtime/vdata_stream_recvr.h
@@ -55,8 +55,7 @@ class MemTrackerLimiter;
class RuntimeState;
namespace pipeline {
-struct ExchangeDataDependency;
-class LocalExchangeChannelDependency;
+class Dependency;
class ExchangeLocalState;
} // namespace pipeline
@@ -115,8 +114,7 @@ public:
bool is_closed() const { return _is_closed; }
- std::shared_ptr<pipeline::LocalExchangeChannelDependency> get_local_channel_dependency(
- int sender_id);
+ std::shared_ptr<pipeline::Dependency> get_local_channel_dependency(int sender_id);
private:
void update_blocks_memory_usage(int64_t size);
@@ -172,8 +170,7 @@ private:
RuntimeProfile::Counter* _blocks_produced_counter = nullptr;
bool _enable_pipeline;
- std::vector<std::shared_ptr<pipeline::LocalExchangeChannelDependency>>
- _sender_to_local_channel_dependency;
+ std::vector<std::shared_ptr<pipeline::Dependency>> _sender_to_local_channel_dependency;
};
class ThreadClosure : public google::protobuf::Closure {
@@ -192,7 +189,7 @@ public:
virtual ~SenderQueue();
void set_local_channel_dependency(
- std::shared_ptr<pipeline::LocalExchangeChannelDependency> local_channel_dependency) {
+ std::shared_ptr<pipeline::Dependency> local_channel_dependency) {
_local_channel_dependency = local_channel_dependency;
}
@@ -216,13 +213,12 @@ public:
return _block_queue.empty();
}
- void set_dependency(std::shared_ptr<pipeline::ExchangeDataDependency> dependency) {
+ void set_dependency(std::shared_ptr<pipeline::Dependency> dependency) {
_dependency = dependency;
}
protected:
friend class pipeline::ExchangeLocalState;
- friend struct pipeline::ExchangeDataDependency;
Status _inner_get_batch_without_lock(Block* block, bool* eos);
void try_set_dep_ready_without_lock();
@@ -289,8 +285,8 @@ protected:
std::deque<std::pair<google::protobuf::Closure*, MonotonicStopWatch>> _pending_closures;
std::unordered_map<std::thread::id, std::unique_ptr<ThreadClosure>> _local_closure;
- std::shared_ptr<pipeline::ExchangeDataDependency> _dependency;
- std::shared_ptr<pipeline::LocalExchangeChannelDependency> _local_channel_dependency;
+ std::shared_ptr<pipeline::Dependency> _dependency;
+ std::shared_ptr<pipeline::Dependency> _local_channel_dependency;
};
class VDataStreamRecvr::PipSenderQueue : public SenderQueue {
diff --git a/be/src/vec/sink/vdata_stream_sender.cpp b/be/src/vec/sink/vdata_stream_sender.cpp
index 35900964274..e690c8eb591 100644
--- a/be/src/vec/sink/vdata_stream_sender.cpp
+++ b/be/src/vec/sink/vdata_stream_sender.cpp
@@ -107,8 +107,7 @@ Status Channel<Parent>::init(RuntimeState* state) {
}
template <typename Parent>
-std::shared_ptr<pipeline::LocalExchangeChannelDependency>
-PipChannel<Parent>::get_local_channel_dependency() {
+std::shared_ptr<pipeline::Dependency> PipChannel<Parent>::get_local_channel_dependency() {
if (!Channel<Parent>::_local_recvr) {
if constexpr (std::is_same_v<pipeline::ExchangeSinkLocalState, Parent>) {
throw Exception(ErrorCode::INTERNAL_ERROR,
diff --git a/be/src/vec/sink/vdata_stream_sender.h b/be/src/vec/sink/vdata_stream_sender.h
index 1224ded31ee..2834c7e2aa5 100644
--- a/be/src/vec/sink/vdata_stream_sender.h
+++ b/be/src/vec/sink/vdata_stream_sender.h
@@ -65,7 +65,7 @@ enum CompressionTypePB : int;
namespace pipeline {
class ExchangeSinkOperator;
class ExchangeSinkOperatorX;
-class LocalExchangeChannelDependency;
+class Dependency;
} // namespace pipeline
namespace vectorized {
@@ -534,7 +534,7 @@ public:
return _send_callback;
}
- std::shared_ptr<pipeline::LocalExchangeChannelDependency> get_local_channel_dependency();
+ std::shared_ptr<pipeline::Dependency> get_local_channel_dependency();
private:
friend class VDataStreamSender;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 17/19: [Improve](InPredict) enhance in predict with struct type (#30840)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ad07dec0ed5a36985d9d747d34a153357c1a90ec
Author: amory <wa...@selectdb.com>
AuthorDate: Thu Feb 22 11:25:06 2024 +0800
[Improve](InPredict) enhance in predict with struct type (#30840)
---
be/src/vec/exprs/vin_predicate.cpp | 3 +
be/src/vec/functions/function_struct_in.cpp | 31 ++++
be/src/vec/functions/function_struct_in.h | 169 +++++++++++++++++++++
be/src/vec/functions/in.h | 3 +-
be/src/vec/functions/simple_function_factory.h | 2 +
be/test/vec/columns/column_hash_func_test.cpp | 26 ++++
.../nereids/trees/expressions/InPredicate.java | 10 ++
.../doris/nereids/util/TypeCoercionUtils.java | 11 +-
.../nereids_syntax_p0/inpredicate_with_struct.out | 19 +++
.../inpredicate_with_struct.groovy | 41 +++++
10 files changed, 311 insertions(+), 4 deletions(-)
diff --git a/be/src/vec/exprs/vin_predicate.cpp b/be/src/vec/exprs/vin_predicate.cpp
index 896b2a903d3..6f57828ef2d 100644
--- a/be/src/vec/exprs/vin_predicate.cpp
+++ b/be/src/vec/exprs/vin_predicate.cpp
@@ -66,6 +66,9 @@ Status VInPredicate::prepare(RuntimeState* state, const RowDescriptor& desc,
// construct the proper function_name
std::string head(_is_not_in ? "not_" : "");
std::string real_function_name = head + std::string(function_name);
+ if (is_struct(remove_nullable(argument_template[0].type))) {
+ real_function_name = "struct_" + real_function_name;
+ }
_function = SimpleFunctionFactory::instance().get_function(real_function_name,
argument_template, _data_type);
if (_function == nullptr) {
diff --git a/be/src/vec/functions/function_struct_in.cpp b/be/src/vec/functions/function_struct_in.cpp
new file mode 100644
index 00000000000..943e3a80f47
--- /dev/null
+++ b/be/src/vec/functions/function_struct_in.cpp
@@ -0,0 +1,31 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// and modified by Doris
+
+#include "vec/functions/function_struct_in.h"
+
+#include "vec/functions/simple_function_factory.h"
+
+namespace doris::vectorized {
+
+void register_function_struct_in(SimpleFunctionFactory& factory) {
+ factory.register_function<FunctionStructIn<false>>();
+ factory.register_function<FunctionStructIn<true>>();
+}
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/functions/function_struct_in.h b/be/src/vec/functions/function_struct_in.h
new file mode 100644
index 00000000000..13ca2b1feca
--- /dev/null
+++ b/be/src/vec/functions/function_struct_in.h
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+
+#pragma once
+
+#include <glog/logging.h>
+#include <stddef.h>
+
+#include <algorithm>
+#include <memory>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "vec/columns/column.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_struct.h"
+#include "vec/columns/column_vector.h"
+#include "vec/columns/columns_number.h"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type_factory.hpp"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/functions/function.h"
+
+namespace doris::vectorized {
+struct ColumnRowRef {
+ ColumnPtr column;
+ size_t row_idx;
+
+ // equals when call set insert, this operator will be used
+ bool operator==(const ColumnRowRef& other) const {
+ return column->compare_at(row_idx, other.row_idx, *column, 0) == 0;
+ }
+ // compare
+ bool operator<(const ColumnRowRef& other) const {
+ return column->compare_at(row_idx, other.row_idx, *column, 0) < 0;
+ }
+
+ // when call set find, will use hash to find
+ size_t operator()(const ColumnRowRef& a) const {
+ uint32_t hash_val = 0;
+ a.column->update_crc_with_value(a.row_idx, a.row_idx + 1, hash_val, nullptr);
+ return hash_val;
+ }
+};
+
+template <bool negative>
+class FunctionStructIn : public IFunction {
+public:
+ static constexpr auto name = negative ? "struct_not_in" : "struct_in";
+
+ static FunctionPtr create() { return std::make_shared<FunctionStructIn>(); }
+
+ String get_name() const override { return name; }
+
+ bool is_variadic() const override { return true; }
+
+ size_t get_number_of_arguments() const override { return 0; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& args) const override {
+ for (const auto& arg : args) {
+ if (arg->is_nullable()) {
+ return make_nullable(std::make_shared<DataTypeUInt8>());
+ }
+ }
+ return std::make_shared<DataTypeUInt8>();
+ }
+
+ bool use_default_implementation_for_nulls() const override { return false; }
+
+ // make data in context into a set
+ Status open(FunctionContext* context, FunctionContext::FunctionStateScope scope) override {
+ DCHECK(context->get_num_args() >= 1);
+ auto* col_desc = context->get_arg_type(0);
+ DataTypePtr args_type = DataTypeFactory::instance().create_data_type(*col_desc);
+ MutableColumnPtr column_struct_ptr_args = remove_nullable(args_type)->create_column();
+ NullMap null_map(context->get_num_args(), false);
+ for (int i = 1; i < context->get_num_args(); ++i) {
+ // FE should make element type consistent and
+ // equalize the length of the elements in struct
+ const auto& const_column_ptr = context->get_constant_col(i);
+ if (const_column_ptr == nullptr) {
+ break;
+ }
+ const auto& [col, _] = unpack_if_const(const_column_ptr->column_ptr);
+ if (col->is_nullable()) {
+ auto* null_col = vectorized::check_and_get_column<vectorized::ColumnNullable>(col);
+ if (null_col->has_null()) {
+ null_in_set = true;
+ null_map[i - 1] = true;
+ } else {
+ column_struct_ptr_args->insert_from(null_col->get_nested_column(), 0);
+ }
+ } else {
+ column_struct_ptr_args->insert_from(*col, 0);
+ }
+ }
+ ColumnPtr column_ptr = std::move(column_struct_ptr_args);
+ // make StructRef into set
+ for (size_t i = 1; i < context->get_num_args(); ++i) {
+ if (null_in_set && null_map[i - 1]) {
+ continue;
+ }
+ args_set.insert({column_ptr, i - 1});
+ }
+ return Status::OK();
+ }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ size_t result, size_t input_rows_count) const override {
+ auto res = ColumnUInt8::create();
+ ColumnUInt8::Container& vec_res = res->get_data();
+ vec_res.resize(input_rows_count);
+
+ ColumnUInt8::MutablePtr col_null_map_to;
+ col_null_map_to = ColumnUInt8::create(input_rows_count, false);
+ auto& vec_null_map_to = col_null_map_to->get_data();
+
+ const ColumnWithTypeAndName& left_arg = block.get_by_position(arguments[0]);
+ const auto& [materialized_column, col_const] = unpack_if_const(left_arg.column);
+
+ for (size_t i = 0; i < input_rows_count; ++i) {
+ ColumnRowRef ref({materialized_column, i});
+ bool find = args_set.find({materialized_column, i}) != args_set.end();
+ if constexpr (negative) {
+ vec_res[i] = !find;
+ } else {
+ vec_res[i] = find;
+ }
+ if (null_in_set) {
+ vec_null_map_to[i] = negative == vec_res[i];
+ } else {
+ vec_null_map_to[i] = false;
+ }
+ }
+
+ if (block.get_by_position(result).type->is_nullable()) {
+ block.replace_by_position(
+ result, ColumnNullable::create(std::move(res), std::move(col_null_map_to)));
+ } else {
+ block.replace_by_position(result, std::move(res));
+ }
+ return Status::OK();
+ }
+
+private:
+ std::unordered_set<ColumnRowRef, ColumnRowRef> args_set;
+ bool null_in_set = false;
+};
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/functions/in.h b/be/src/vec/functions/in.h
index 42cb5be6166..9fa182caf03 100644
--- a/be/src/vec/functions/in.h
+++ b/be/src/vec/functions/in.h
@@ -125,7 +125,6 @@ public:
state->hybrid_set.reset(
create_set(context->get_arg_type(0)->type, get_size_with_out_null(context)));
}
-
for (int i = 1; i < context->get_num_args(); ++i) {
const auto& const_column_ptr = context->get_constant_col(i);
if (const_column_ptr != nullptr) {
@@ -193,7 +192,7 @@ public:
}
} else { // non-nullable
- if (materialized_column->is_column_string()) {
+ if (WhichDataType(left_arg.type).is_string()) {
const auto* column_string_ptr =
assert_cast<const vectorized::ColumnString*>(materialized_column.get());
search_hash_set(in_state, input_rows_count, vec_res, column_string_ptr);
diff --git a/be/src/vec/functions/simple_function_factory.h b/be/src/vec/functions/simple_function_factory.h
index 9bedc204cb0..b1c1b394bff 100644
--- a/be/src/vec/functions/simple_function_factory.h
+++ b/be/src/vec/functions/simple_function_factory.h
@@ -65,6 +65,7 @@ void register_function_running_difference(SimpleFunctionFactory& factory);
void register_function_date_time_to_string(SimpleFunctionFactory& factory);
void register_function_date_time_string_to_string(SimpleFunctionFactory& factory);
void register_function_in(SimpleFunctionFactory& factory);
+void register_function_struct_in(SimpleFunctionFactory& factory);
void register_function_if(SimpleFunctionFactory& factory);
void register_function_nullif(SimpleFunctionFactory& factory);
void register_function_date_time_computation(SimpleFunctionFactory& factory);
@@ -245,6 +246,7 @@ public:
register_function_time_of_function(instance);
register_function_string(instance);
register_function_in(instance);
+ register_function_struct_in(instance);
register_function_if(instance);
register_function_nullif(instance);
register_function_date_time_computation(instance);
diff --git a/be/test/vec/columns/column_hash_func_test.cpp b/be/test/vec/columns/column_hash_func_test.cpp
index 0a9471a1ab3..7b2d5f2dddd 100644
--- a/be/test/vec/columns/column_hash_func_test.cpp
+++ b/be/test/vec/columns/column_hash_func_test.cpp
@@ -233,4 +233,30 @@ TEST(HashFuncTest, StructTypeTest) {
std::cout << crc_hashes[0] << std::endl;
}
+TEST(HashFuncTest, StructTypeTestWithSepcificValueCrcHash) {
+ DataTypePtr n1 = std::make_shared<DataTypeInt64>();
+ DataTypePtr s1 = std::make_shared<DataTypeString>();
+ DataTypes dataTypes;
+ dataTypes.push_back(n1);
+ dataTypes.push_back(s1);
+
+ Tuple t;
+ t.push_back(Int64(1));
+ t.push_back(String("hello"));
+
+ DataTypePtr a = std::make_shared<DataTypeStruct>(dataTypes);
+ std::cout << a->get_name() << std::endl;
+ MutableColumnPtr struct_mutable_col = a->create_column();
+ struct_mutable_col->insert(t);
+
+ uint32_t hash_val = 0;
+ struct_mutable_col->update_crc_with_value(0, 1, hash_val, nullptr);
+
+ for (int i = 0; i < 100; ++i) {
+ uint32_t should_same_hash_val = 0;
+ struct_mutable_col->update_crc_with_value(0, 1, should_same_hash_val, nullptr);
+ EXPECT_EQ(hash_val, should_same_hash_val);
+ }
+}
+
} // namespace doris::vectorized
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/InPredicate.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/InPredicate.java
index c86a074dcfd..f3ae9ce5b27 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/InPredicate.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/InPredicate.java
@@ -76,6 +76,16 @@ public class InPredicate extends Expression {
@Override
public void checkLegalityBeforeTypeCoercion() {
+ if (children().get(0).getDataType().isStructType()) {
+ // we should check in value list is all struct type
+ for (int i = 1; i < children().size(); i++) {
+ if (!children().get(i).getDataType().isStructType() && !children().get(i).getDataType().isNullType()) {
+ throw new AnalysisException("in predicate struct should compare with struct type list, but got : "
+ + children().get(i).getDataType().toSql());
+ }
+ }
+ return;
+ }
children().forEach(c -> {
if (c.getDataType().isObjectType()) {
throw new AnalysisException("in predicate could not contains object type: " + this.toSql());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
index a3d4a84d3cb..239972fc1d6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
@@ -953,7 +953,8 @@ public class TypeCoercionUtils {
if (inPredicate.getOptions().stream().map(Expression::getDataType)
.allMatch(dt -> dt.equals(inPredicate.getCompareExpr().getDataType()))) {
- if (!supportCompare(inPredicate.getCompareExpr().getDataType())) {
+ if (!supportCompare(inPredicate.getCompareExpr().getDataType())
+ && !inPredicate.getCompareExpr().getDataType().isStructType()) {
throw new AnalysisException("data type " + inPredicate.getCompareExpr().getDataType()
+ " could not used in InPredicate " + inPredicate.toSql());
}
@@ -964,7 +965,13 @@ public class TypeCoercionUtils {
.stream()
.map(Expression::getDataType).collect(Collectors.toList()),
true);
- if (optionalCommonType.isPresent() && !supportCompare(optionalCommonType.get())) {
+ if (inPredicate.getCompareExpr().getDataType().isStructType() && optionalCommonType.isPresent()
+ && !optionalCommonType.get().isStructType()) {
+ throw new AnalysisException("data type " + optionalCommonType.get()
+ + " is not match " + inPredicate.getCompareExpr().getDataType() + " used in InPredicate");
+ }
+ if (optionalCommonType.isPresent() && !supportCompare(optionalCommonType.get())
+ && !optionalCommonType.get().isStructType()) {
throw new AnalysisException("data type " + optionalCommonType.get()
+ " could not used in InPredicate " + inPredicate.toSql());
}
diff --git a/regression-test/data/nereids_syntax_p0/inpredicate_with_struct.out b/regression-test/data/nereids_syntax_p0/inpredicate_with_struct.out
new file mode 100644
index 00000000000..5e1e97a235b
--- /dev/null
+++ b/regression-test/data/nereids_syntax_p0/inpredicate_with_struct.out
@@ -0,0 +1,19 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !in_predicate_11 --
+\N
+
+-- !in_predicate_12 --
+\N
+
+-- !in_predicate_13 --
+true
+
+-- !in_predicate_14 --
+false
+
+-- !in_predicate_15 --
+true
+
+-- !in_predicate_16 --
+false
+
diff --git a/regression-test/suites/nereids_syntax_p0/inpredicate_with_struct.groovy b/regression-test/suites/nereids_syntax_p0/inpredicate_with_struct.groovy
new file mode 100644
index 00000000000..fb686fbc9ef
--- /dev/null
+++ b/regression-test/suites/nereids_syntax_p0/inpredicate_with_struct.groovy
@@ -0,0 +1,41 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("inpredicate_with_struct") {
+ sql """ set enable_nereids_planner = true;"""
+ sql """ set enable_fallback_to_original_planner=false;"""
+ // support struct type
+ order_qt_in_predicate_11 """
+ select struct(1,"2") in (struct(1,3), null);
+ """
+ order_qt_in_predicate_12 """
+ select struct(1,"2") not in (struct(1,3), null);
+ """
+ order_qt_in_predicate_13 """
+ select struct(1,"2") in (struct(1,3), struct(1,2));
+ """
+ order_qt_in_predicate_14 """
+ select struct(1,"2") not in (struct(1,3), struct(1,2));
+ """
+ order_qt_in_predicate_15 """
+ select struct(1,"2") in (struct(1,3), struct(1,"2"), struct(1,1));
+ """
+ order_qt_in_predicate_16 """
+ select struct(1,"2") not in (struct(1,3), struct(1,"2"), struct(1,1));
+ """
+}
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 13/19: [doc](create table) add since version for some new features (#30506)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit bb73561b71264e3d8c62e8401c3873c43b28a5d1
Author: zhannngchen <48...@users.noreply.github.com>
AuthorDate: Thu Feb 22 09:13:35 2024 +0800
[doc](create table) add since version for some new features (#30506)
---
.../sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md | 4 ++--
.../sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
index 012720428c2..44dbf5cceb8 100644
--- a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
+++ b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
@@ -115,7 +115,7 @@ Column definition list:
BITMAP_UNION: The aggregation mode of BIMTAP type columns, which performs the union aggregation of bitmaps.
```
- * `AUTO_INCREMENT(auto_inc_start_value)`(Available from version 2.1 and later.)
+ * <version since="2.1" type="inline">`AUTO_INCREMENT(auto_inc_start_value)`</version>
To indicate if the column is a auto-increment column. Auto-increment column can be used to generate a unique identity for new row. If no values are assgined for auto-increment column when inserting, Doris will generate sequence numbers automatically. You can also assign the auto-increment column with NULL literal to indicate Doris to generate sequence numbers. It should be noted that, for performance reasons, BE will cache some values of auto-increment column in memory. Therefore [...]
A table can have at most one auto-incremnt column. The auto-increment column should be BIGINT type and be NOT NULL.
@@ -143,7 +143,7 @@ Column definition list:
dt DATETIME DEFAULT CURRENT_TIMESTAMP
```
- * `on update current_timestamp`
+ * <version since="2.1" type="inline">`on update current_timestamp`</version>
To indicate that whether the value of this column should be updated to the current timestamp (`current_timestamp`) when there is an update on the row. The feature is only available on unique table with merge-on-write enabled. Columns with this feature enabled must declare a default value, and the default value must be `current_timestamp`. If the precision of the timestamp is declared here, the timestamp precision in the default value of the column must be the same as the precisio [...]
diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
index e760bf9892f..e8ac8e69f0a 100644
--- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
+++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
@@ -106,7 +106,7 @@ distribution_desc
HLL_UNION：HLL 类型的列的聚合方式，通过 HyperLogLog 算法聚合。
BITMAP_UNION：BIMTAP 类型的列的聚合方式，进行位图的并集聚合。
```
- * `AUTO_INCREMENT(auto_inc_start_value)`(2.1版本及以后可用)
+ * <version since="2.1" type="inline">`AUTO_INCREMENT(auto_inc_start_value)`</version>
是否为自增列，自增列可以用来为新插入的行生成一个唯一标识。在插入表数据时如果没有指定自增列的值，则会自动生成一个合法的值。当自增列被显示地插入NULL时，其值也会被替换为生成的合法值。需要注意的是，处于性能考虑，BE会在内存中缓存部分自增列的值，所以自增列自动生成的值只能保证单调性和唯一性，无法保证严格的连续性。
一张表中至多有一个列是自增列，自增列必须是BIGINT类型，且必须为NOT NULL。
@@ -130,7 +130,7 @@ distribution_desc
// 只用于DATETIME类型，导入数据缺失该值时系统将赋予当前时间
dt DATETIME DEFAULT CURRENT_TIMESTAMP
```
- * `on update current_timestamp`
+ * <version since="2.1" type="inline">`on update current_timestamp`</version>
是否在该行有列更新时将该列的值更新为当前时间(`current_timestamp`)。该特性只能在开启了merge-on-write的unique表上使用，开启了这个特性的列必须声明默认值，且默认值必须为`current_timestamp`。如果此处声明了时间戳的精度，则该列默认值中的时间戳精度必须与该处的时间戳精度相同。
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 01/19: Example Modify audit log documents (#31192)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 57f3ea6d7bf6dc878b7dcab5f636342cc2eb06d6
Author: yuanyuan8983 <99...@users.noreply.github.com>
AuthorDate: Wed Feb 21 19:43:40 2024 +0800
Example Modify audit log documents (#31192)
---
docs/en/docs/ecosystem/audit-plugin.md | 2 +-
docs/zh-CN/docs/ecosystem/audit-plugin.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/en/docs/ecosystem/audit-plugin.md b/docs/en/docs/ecosystem/audit-plugin.md
index fc8e7560d10..5b3f5d3be11 100644
--- a/docs/en/docs/ecosystem/audit-plugin.md
+++ b/docs/en/docs/ecosystem/audit-plugin.md
@@ -102,7 +102,7 @@ After executing install, the AuditLoader directory will be automatically generat
* database: Audit log database name.
* audit_log_table: Audit log table name.
* slow_log_table: Slow query log table name.
- * enable_slow_log: Whether to enable the slow query log import function. The default value is false.
+ * enable_slow_log: Whether to enable the slow query log import function. The default value is false. You can set the slow query threshold in the FE configuration item. The parameter is qe_slow_log_ms and the default value is 5s.
* user: Cluster username. The user must have INSERT permission on the corresponding table.
* password: Cluster user password.
diff --git a/docs/zh-CN/docs/ecosystem/audit-plugin.md b/docs/zh-CN/docs/ecosystem/audit-plugin.md
index a6fd56c3468..6ea53903346 100644
--- a/docs/zh-CN/docs/ecosystem/audit-plugin.md
+++ b/docs/zh-CN/docs/ecosystem/audit-plugin.md
@@ -105,7 +105,7 @@ Doris 2.1 版本之前的用户，请参阅如下方式使用审计日志插件
* database：审计日志库名。
* audit_log_table：审计日志表名。
* slow_log_table：慢查询日志表名。
- * enable_slow_log：是否开启慢查询日志导入功能。默认值为 false。
+ * enable_slow_log：是否开启慢查询日志导入功能。默认值为 false。可以在FE配置项中配置慢查询的阈值，参数为 qe_slow_log_ms ，默认5s。
* user：集群用户名。该用户必须具有对应表的 INSERT 权限。
* password：集群用户密码。
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 11/19: [log](query) add debug log for check query visible version (#31217)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit d080fe8c89a20962134ca119e3b6f964c1eff5e7
Author: HHoflittlefish777 <77...@users.noreply.github.com>
AuthorDate: Thu Feb 22 09:01:22 2024 +0800
[log](query) add debug log for check query visible version (#31217)
---
fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
index 9f975a6b0a2..54cec43ea4f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
@@ -61,6 +61,7 @@ import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.DebugUtil;
import org.apache.doris.common.util.Util;
import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
import org.apache.doris.qe.ConnectContext;
@@ -747,6 +748,10 @@ public class OlapScanNode extends ScanNode {
useFixReplica = ConnectContext.get().getSessionVariable().useFixReplica;
// if use_fix_replica is set to true, set skip_missing_version to false
skipMissingVersion = useFixReplica == -1 && ConnectContext.get().getSessionVariable().skipMissingVersion;
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("query id: {}, partition id:{} visibleVersion: {}",
+ DebugUtil.printId(ConnectContext.get().queryId()), partition.getId(), visibleVersion);
+ }
}
for (Tablet tablet : tablets) {
long tabletId = tablet.getId();
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 10/19: Window function sum/avg does not support distinct document modification (#31066)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6ef90b2ad96ee7c3f37cef57442f48f752a3538a
Author: caoliang-web <71...@users.noreply.github.com>
AuthorDate: Thu Feb 22 00:02:00 2024 +0800
Window function sum/avg does not support distinct document modification (#31066)
---
.../sql-manual/sql-functions/window-functions/window-function-avg.md | 2 +-
.../sql-manual/sql-functions/window-functions/window-function-sum.md | 2 +-
.../sql-manual/sql-functions/window-functions/window-function-avg.md | 2 +-
.../sql-manual/sql-functions/window-functions/window-function-sum.md | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/docs/en/docs/sql-manual/sql-functions/window-functions/window-function-avg.md b/docs/en/docs/sql-manual/sql-functions/window-functions/window-function-avg.md
index b7626b64d10..22a48eee387 100644
--- a/docs/en/docs/sql-manual/sql-functions/window-functions/window-function-avg.md
+++ b/docs/en/docs/sql-manual/sql-functions/window-functions/window-function-avg.md
@@ -17,7 +17,7 @@ Unless required by applicable law or agreed to in writing, software distributed
Calculate the mean of the data within the window
```sql
-AVG([DISTINCT | ALL] *expression*) [OVER (*analytic_clause*)]
+AVG([ALL] *expression*) [OVER (*analytic_clause*)]
```
### example
diff --git a/docs/en/docs/sql-manual/sql-functions/window-functions/window-function-sum.md b/docs/en/docs/sql-manual/sql-functions/window-functions/window-function-sum.md
index 11d29c401df..cd75600e3d9 100644
--- a/docs/en/docs/sql-manual/sql-functions/window-functions/window-function-sum.md
+++ b/docs/en/docs/sql-manual/sql-functions/window-functions/window-function-sum.md
@@ -17,7 +17,7 @@ Unless required by applicable law or agreed to in writing, software distributed
Calculate the sum of the data in the window
```sql
-SUM([DISTINCT | ALL] expression) [OVER (analytic_clause)]
+SUM([ALL] expression) [OVER (analytic_clause)]
```
### example
diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/window-functions/window-function-avg.md b/docs/zh-CN/docs/sql-manual/sql-functions/window-functions/window-function-avg.md
index 426df16d7b5..0868f1672e1 100644
--- a/docs/zh-CN/docs/sql-manual/sql-functions/window-functions/window-function-avg.md
+++ b/docs/zh-CN/docs/sql-manual/sql-functions/window-functions/window-function-avg.md
@@ -17,7 +17,7 @@ Unless required by applicable law or agreed to in writing, software distributed
计算窗口内数据的平均值
```sql
-AVG([DISTINCT | ALL] *expression*) [OVER (*analytic_clause*)]
+AVG([ALL] *expression*) [OVER (*analytic_clause*)]
```
### example
diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/window-functions/window-function-sum.md b/docs/zh-CN/docs/sql-manual/sql-functions/window-functions/window-function-sum.md
index b5e71e11839..22bcc7f972d 100644
--- a/docs/zh-CN/docs/sql-manual/sql-functions/window-functions/window-function-sum.md
+++ b/docs/zh-CN/docs/sql-manual/sql-functions/window-functions/window-function-sum.md
@@ -17,7 +17,7 @@ Unless required by applicable law or agreed to in writing, software distributed
计算窗口内数据的和
```sql
-SUM([DISTINCT | ALL] expression) [OVER (analytic_clause)]
+SUM([ALL] expression) [OVER (analytic_clause)]
```
### example
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 12/19: [fix](group_commit)Fix bound checking problem when reading wal block (#31112)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b66583551c331f8dc37bb854f3f74cbb5bf1cb8d
Author: huanghaibin <28...@qq.com>
AuthorDate: Thu Feb 22 09:09:35 2024 +0800
[fix](group_commit)Fix bound checking problem when reading wal block (#31112)
---
be/src/vec/exec/format/wal/wal_reader.cpp | 21 ++--
be/test/vec/exec/vwal_scanner_test.cpp | 155 ++++++++++++++++++------------
2 files changed, 106 insertions(+), 70 deletions(-)
diff --git a/be/src/vec/exec/format/wal/wal_reader.cpp b/be/src/vec/exec/format/wal/wal_reader.cpp
index cb89dd9bcf0..981e740d05c 100644
--- a/be/src/vec/exec/format/wal/wal_reader.cpp
+++ b/be/src/vec/exec/format/wal/wal_reader.cpp
@@ -18,6 +18,7 @@
#include "wal_reader.h"
#include "common/logging.h"
+#include "common/sync_point.h"
#include "gutil/strings/split.h"
#include "olap/wal/wal_manager.h"
#include "runtime/runtime_state.h"
@@ -61,11 +62,17 @@ Status WalReader::get_next_block(Block* block, size_t* read_rows, bool* eof) {
//convert to dst block
vectorized::Block dst_block;
int index = 0;
- auto columns = block->get_columns_with_type_and_name();
- if (_column_id_count != columns.size() || columns.size() != _tuple_descriptor->slots().size()) {
+ auto output_block_columns = block->get_columns_with_type_and_name();
+ size_t output_block_column_size = output_block_columns.size();
+ TEST_SYNC_POINT_CALLBACK("WalReader::set_column_id_count", &_column_id_count);
+ TEST_SYNC_POINT_CALLBACK("WalReader::set_out_block_column_size", &output_block_column_size);
+ if (_column_id_count != src_block.columns() ||
+ output_block_column_size != _tuple_descriptor->slots().size()) {
return Status::InternalError(
- "not equal _column_id_count={} vs columns size={} vs tuple_descriptor size={}",
- std::to_string(_column_id_count), std::to_string(columns.size()),
+ "not equal wal _column_id_count={} vs wal block columns size={}, "
+ "output block columns size={} vs tuple_descriptor size={}",
+ std::to_string(_column_id_count), std::to_string(src_block.columns()),
+ std::to_string(output_block_column_size),
std::to_string(_tuple_descriptor->slots().size()));
}
for (auto slot_desc : _tuple_descriptor->slots()) {
@@ -78,9 +85,9 @@ Status WalReader::get_next_block(Block* block, size_t* read_rows, bool* eof) {
if (column_ptr != nullptr && slot_desc->is_nullable()) {
column_ptr = make_nullable(column_ptr);
}
- dst_block.insert(
- index, vectorized::ColumnWithTypeAndName(std::move(column_ptr), columns[index].type,
- columns[index].name));
+ dst_block.insert(index, vectorized::ColumnWithTypeAndName(
+ std::move(column_ptr), output_block_columns[index].type,
+ output_block_columns[index].name));
index++;
}
block->swap(dst_block);
diff --git a/be/test/vec/exec/vwal_scanner_test.cpp b/be/test/vec/exec/vwal_scanner_test.cpp
index 8d4fe6ad75f..af292640368 100644
--- a/be/test/vec/exec/vwal_scanner_test.cpp
+++ b/be/test/vec/exec/vwal_scanner_test.cpp
@@ -20,6 +20,7 @@
#include <vector>
#include "common/object_pool.h"
+#include "common/sync_point.h"
#include "gen_cpp/Descriptors_types.h"
#include "gen_cpp/PlanNodes_types.h"
#include "io/fs/local_file_system.h"
@@ -40,12 +41,16 @@ public:
init();
_profile = _runtime_state.runtime_profile();
_runtime_state.init_mem_trackers();
- static_cast<void>(_runtime_state.init(unique_id, query_options, query_globals, _env));
+ WARN_IF_ERROR(_runtime_state.init(_unique_id, _query_options, _query_globals, _env),
+ "fail to init _runtime_state");
}
void init();
+ void generate_scanner(std::shared_ptr<VFileScanner>& scanner);
void TearDown() override {
- static_cast<void>(io::global_local_filesystem()->delete_directory(wal_dir));
+ WARN_IF_ERROR(_scan_node->close(&_runtime_state), "fail to close scan_node")
+ WARN_IF_ERROR(io::global_local_filesystem()->delete_directory(_wal_dir),
+ fmt::format("fail to delete dir={}", _wal_dir));
SAFE_STOP(_env->_wal_manager);
}
@@ -53,16 +58,16 @@ protected:
virtual void SetUp() override {}
private:
- void init_desc_table();
+ void _init_desc_table();
ExecEnv* _env = nullptr;
- std::string wal_dir = std::string(getenv("DORIS_HOME")) + "/wal_test";
- int64_t db_id = 1;
- int64_t tb_id = 2;
- int64_t txn_id = 789;
- int64_t version = 0;
- int64_t backend_id = 1001;
- std::string label = "test";
+ std::string _wal_dir = std::string(getenv("DORIS_HOME")) + "/wal_test";
+ int64_t _db_id = 1;
+ int64_t _tb_id = 2;
+ int64_t _txn_id = 789;
+ int64_t _version = 0;
+ int64_t _backend_id = 1001;
+ std::string _label = "test";
TupleId _dst_tuple_id = 0;
RuntimeState _runtime_state;
@@ -73,12 +78,18 @@ private:
ScannerCounter _counter;
std::vector<TExpr> _pre_filter;
TPlanNode _tnode;
- TUniqueId unique_id;
- TQueryOptions query_options;
- TQueryGlobals query_globals;
+ TUniqueId _unique_id;
+ TQueryOptions _query_options;
+ TQueryGlobals _query_globals;
+ std::shared_ptr<NewFileScanNode> _scan_node = nullptr;
+ std::vector<TFileRangeDesc> _ranges;
+ TFileRangeDesc _range_desc;
+ TFileScanRange _scan_range;
+ std::unique_ptr<ShardedKVCache> _kv_cache = nullptr;
+ std::unique_ptr<TMasterInfo> _master_info = nullptr;
};
-void VWalScannerTest::init_desc_table() {
+void VWalScannerTest::_init_desc_table() {
TDescriptorTable t_desc_table;
// table descriptors
@@ -118,6 +129,7 @@ void VWalScannerTest::init_desc_table() {
slot_desc.nullIndicatorBit = -1;
slot_desc.colName = "c1";
slot_desc.slotIdx = 1;
+ slot_desc.col_unique_id = 0;
slot_desc.isMaterialized = true;
t_desc_table.slotDescriptors.push_back(slot_desc);
@@ -145,6 +157,7 @@ void VWalScannerTest::init_desc_table() {
slot_desc.nullIndicatorBit = -1;
slot_desc.colName = "c2";
slot_desc.slotIdx = 2;
+ slot_desc.col_unique_id = 1;
slot_desc.isMaterialized = true;
t_desc_table.slotDescriptors.push_back(slot_desc);
@@ -172,6 +185,7 @@ void VWalScannerTest::init_desc_table() {
slot_desc.nullIndicatorBit = -1;
slot_desc.colName = "c3";
slot_desc.slotIdx = 3;
+ slot_desc.col_unique_id = 2;
slot_desc.isMaterialized = true;
t_desc_table.slotDescriptors.push_back(slot_desc);
@@ -196,9 +210,10 @@ void VWalScannerTest::init_desc_table() {
void VWalScannerTest::init() {
config::group_commit_wal_max_disk_limit = "100M";
- init_desc_table();
- static_cast<void>(io::global_local_filesystem()->create_directory(
- wal_dir + "/" + std::to_string(db_id) + "/" + std::to_string(tb_id)));
+ _init_desc_table();
+ WARN_IF_ERROR(io::global_local_filesystem()->create_directory(
+ _wal_dir + "/" + std::to_string(_db_id) + "/" + std::to_string(_tb_id)),
+ "fail to creat directory");
// Node Id
_tnode.node_id = 0;
@@ -210,73 +225,87 @@ void VWalScannerTest::init() {
_tnode.file_scan_node.tuple_id = 0;
_tnode.__isset.file_scan_node = true;
+ _scan_node = std::make_shared<NewFileScanNode>(&_obj_pool, _tnode, *_desc_tbl);
+ _scan_node->_output_tuple_desc = _runtime_state.desc_tbl().get_tuple_descriptor(_dst_tuple_id);
+ WARN_IF_ERROR(_scan_node->init(_tnode, &_runtime_state), "fail to init scan_node");
+ WARN_IF_ERROR(_scan_node->prepare(&_runtime_state), "fail to prepare scan_node");
+
+ _range_desc.start_offset = 0;
+ _range_desc.size = 1000;
+ _ranges.push_back(_range_desc);
+ _scan_range.ranges = _ranges;
+ _scan_range.__isset.params = true;
+ _scan_range.params.format_type = TFileFormatType::FORMAT_WAL;
+ _kv_cache.reset(new ShardedKVCache(48));
+
+ _runtime_state._wal_id = _txn_id;
+
+ _master_info.reset(new TMasterInfo());
_env = ExecEnv::GetInstance();
- _env->_master_info = new TMasterInfo();
+ _env->_master_info = _master_info.get();
_env->_master_info->network_address.hostname = "host name";
- _env->_master_info->network_address.port = backend_id;
+ _env->_master_info->network_address.port = _backend_id;
_env->_master_info->backend_id = 1001;
- _env->_wal_manager = WalManager::create_shared(_env, wal_dir);
+ _env->_wal_manager = WalManager::create_shared(_env, _wal_dir);
std::string base_path;
auto st = _env->_wal_manager->_init_wal_dirs_info();
- st = _env->_wal_manager->create_wal_path(db_id, tb_id, txn_id, label, base_path);
+ st = _env->_wal_manager->create_wal_path(_db_id, _tb_id, _txn_id, _label, base_path);
std::string src = "./be/test/exec/test_data/wal_scanner/wal";
- std::string dst = wal_dir + "/" + std::to_string(db_id) + "/" + std::to_string(tb_id) + "/" +
- std::to_string(version) + "_" + std::to_string(backend_id) + "_" +
- std::to_string(txn_id) + "_" + label;
+ std::string dst = _wal_dir + "/" + std::to_string(_db_id) + "/" + std::to_string(_tb_id) + "/" +
+ std::to_string(_version) + "_" + std::to_string(_backend_id) + "_" +
+ std::to_string(_txn_id) + "_" + _label;
std::filesystem::copy(src, dst);
}
-TEST_F(VWalScannerTest, normal) {
- std::vector<size_t> index_vector;
- index_vector.emplace_back(0);
- index_vector.emplace_back(1);
- index_vector.emplace_back(2);
- // config::group_commit_replay_wal_dir = wal_dir;
- NewFileScanNode scan_node(&_obj_pool, _tnode, *_desc_tbl);
- scan_node._output_tuple_desc = _runtime_state.desc_tbl().get_tuple_descriptor(_dst_tuple_id);
- static_cast<void>(scan_node.init(_tnode, &_runtime_state));
- auto status = scan_node.prepare(&_runtime_state);
- EXPECT_TRUE(status.ok());
-
- std::vector<TFileRangeDesc> ranges;
- TFileRangeDesc range_desc;
- {
- range_desc.start_offset = 0;
- range_desc.size = 1000;
- }
- ranges.push_back(range_desc);
- TFileScanRange scan_range;
- scan_range.ranges = ranges;
- scan_range.__isset.params = true;
- scan_range.params.format_type = TFileFormatType::FORMAT_WAL;
- std::unique_ptr<ShardedKVCache> _kv_cache;
- _kv_cache.reset(new ShardedKVCache(48));
- _runtime_state._wal_id = txn_id;
- VFileScanner scanner(&_runtime_state, &scan_node, -1, scan_range, _profile, _kv_cache.get());
- scanner._is_load = false;
+void VWalScannerTest::generate_scanner(std::shared_ptr<VFileScanner>& scanner) {
+ scanner = std::make_shared<VFileScanner>(&_runtime_state, _scan_node.get(), -1, _scan_range,
+ _profile, _kv_cache.get());
+ scanner->_is_load = false;
vectorized::VExprContextSPtrs _conjuncts;
std::unordered_map<std::string, ColumnValueRangeType> _colname_to_value_range;
std::unordered_map<std::string, int> _colname_to_slot_id;
- static_cast<void>(scanner.prepare(_conjuncts, &_colname_to_value_range, &_colname_to_slot_id));
+ WARN_IF_ERROR(scanner->prepare(_conjuncts, &_colname_to_value_range, &_colname_to_slot_id),
+ "fail to prepare scanner");
+}
+TEST_F(VWalScannerTest, normal) {
+ std::shared_ptr<VFileScanner> scanner = nullptr;
+ generate_scanner(scanner);
std::unique_ptr<vectorized::Block> block(new vectorized::Block());
bool eof = false;
- auto st = scanner.get_block(&_runtime_state, block.get(), &eof);
- EXPECT_EQ(3, block->rows());
+ auto st = scanner->get_block(&_runtime_state, block.get(), &eof);
ASSERT_TRUE(st.ok());
+ EXPECT_EQ(3, block->rows());
block->clear();
- st = scanner.get_block(&_runtime_state, block.get(), &eof);
+ st = scanner->get_block(&_runtime_state, block.get(), &eof);
ASSERT_TRUE(st.ok());
EXPECT_EQ(0, block->rows());
ASSERT_TRUE(eof);
- static_cast<void>(scanner.close(&_runtime_state));
- static_cast<void>(scan_node.close(&_runtime_state));
+ WARN_IF_ERROR(scanner->close(&_runtime_state), "fail to close scanner");
+}
- {
- std::stringstream ss;
- scan_node.runtime_profile()->pretty_print(&ss);
- LOG(INFO) << ss.str();
- }
+TEST_F(VWalScannerTest, fail_with_not_equal) {
+ auto sp = SyncPoint::get_instance();
+ Defer defer {[sp] {
+ sp->clear_call_back("WalReader::set_column_id_count");
+ sp->clear_call_back("WalReader::set_out_block_column_size");
+ }};
+ sp->set_call_back("WalReader::set_column_id_count",
+ [](auto&& args) { *try_any_cast<int64_t*>(args[0]) = 2; });
+ sp->set_call_back("WalReader::set_out_block_column_size",
+ [](auto&& args) { *try_any_cast<size_t*>(args[0]) = 2; });
+ sp->enable_processing();
+
+ std::shared_ptr<VFileScanner> scanner = nullptr;
+ generate_scanner(scanner);
+ std::unique_ptr<vectorized::Block> block(new vectorized::Block());
+ bool eof = false;
+ auto st = scanner->get_block(&_runtime_state, block.get(), &eof);
+ ASSERT_FALSE(st.ok());
+ auto msg = st.to_string();
+ auto pos = msg.find("not equal");
+ ASSERT_TRUE(pos != msg.npos);
+ WARN_IF_ERROR(scanner->close(&_runtime_state), "fail to close scanner");
}
} // namespace vectorized
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 02/19: [fix](stream-load) print stream load profile for pipeline and pipelinex (#31198)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e183e9ac460b2c8fa6547fd96e27467954d1489d
Author: Xin Liao <li...@126.com>
AuthorDate: Wed Feb 21 20:01:17 2024 +0800
[fix](stream-load) print stream load profile for pipeline and pipelinex (#31198)
---
be/src/pipeline/pipeline_fragment_context.cpp | 15 +++++++++++++++
.../pipeline/pipeline_x/pipeline_x_fragment_context.cpp | 15 +++++++++++++++
2 files changed, 30 insertions(+)
diff --git a/be/src/pipeline/pipeline_fragment_context.cpp b/be/src/pipeline/pipeline_fragment_context.cpp
index 6c183a96f64..e251b203794 100644
--- a/be/src/pipeline/pipeline_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_fragment_context.cpp
@@ -887,6 +887,21 @@ void PipelineFragmentContext::_close_fragment_instance() {
_runtime_state->runtime_profile()->total_time_counter()->update(
_fragment_watcher.elapsed_time());
static_cast<void>(send_report(true));
+ if (_is_report_success) {
+ std::stringstream ss;
+ // Compute the _local_time_percent before pretty_print the runtime_profile
+ // Before add this operation, the print out like that:
+ // UNION_NODE (id=0):(Active: 56.720us, non-child: 00.00%)
+ // After add the operation, the print out like that:
+ // UNION_NODE (id=0):(Active: 56.720us, non-child: 82.53%)
+ // We can easily know the exec node execute time without child time consumed.
+ _runtime_state->runtime_profile()->compute_time_in_profile();
+ _runtime_state->runtime_profile()->pretty_print(&ss);
+ if (_runtime_state->load_channel_profile()) {
+ _runtime_state->load_channel_profile()->pretty_print(&ss);
+ }
+ LOG(INFO) << ss.str();
+ }
// all submitted tasks done
_exec_env->fragment_mgr()->remove_pipeline_context(
std::dynamic_pointer_cast<PipelineFragmentContext>(shared_from_this()));
diff --git a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
index 38b7b7457fe..9087f94db5e 100644
--- a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -1244,6 +1244,21 @@ void PipelineXFragmentContext::_close_fragment_instance() {
Defer defer_op {[&]() { _is_fragment_instance_closed = true; }};
_runtime_profile->total_time_counter()->update(_fragment_watcher.elapsed_time());
static_cast<void>(send_report(true));
+ if (_is_report_success) {
+ std::stringstream ss;
+ // Compute the _local_time_percent before pretty_print the runtime_profile
+ // Before add this operation, the print out like that:
+ // UNION_NODE (id=0):(Active: 56.720us, non-child: 00.00%)
+ // After add the operation, the print out like that:
+ // UNION_NODE (id=0):(Active: 56.720us, non-child: 82.53%)
+ // We can easily know the exec node execute time without child time consumed.
+ _runtime_state->runtime_profile()->compute_time_in_profile();
+ _runtime_state->runtime_profile()->pretty_print(&ss);
+ if (_runtime_state->load_channel_profile()) {
+ _runtime_state->load_channel_profile()->pretty_print(&ss);
+ }
+ LOG(INFO) << ss.str();
+ }
// all submitted tasks done
_exec_env->fragment_mgr()->remove_pipeline_context(
std::dynamic_pointer_cast<PipelineXFragmentContext>(shared_from_this()));
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 03/19: [fix](Nereids): OuterJoinAssoc reject literal condition (#31172)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2069f9ff63d6e5b5459ab0fb046d07a1238ceb6a
Author: jakevin <ja...@gmail.com>
AuthorDate: Wed Feb 21 20:22:50 2024 +0800
[fix](Nereids): OuterJoinAssoc reject literal condition (#31172)
---
.../org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java | 2 +-
.../doris/nereids/rules/exploration/join/OuterJoinAssocProject.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java
index 2080cfce93d..3208654fa78 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java
@@ -79,7 +79,7 @@ public class OuterJoinAssoc extends OneExplorationRuleFactory {
.addAll(topJoin.getOtherJoinConjuncts()).build();
Set<Slot> notNullSlots = ExpressionUtils.inferNotNullSlots(on,
ctx.cascadesContext);
- if (!conditionSlot.equals(notNullSlots)) {
+ if (conditionSlot.isEmpty() || !conditionSlot.equals(notNullSlots)) {
return null;
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java
index 512aec7bbc7..5cd22058c04 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java
@@ -80,7 +80,7 @@ public class OuterJoinAssocProject extends OneExplorationRuleFactory {
.addAll(topJoin.getOtherJoinConjuncts()).build();
Set<Slot> notNullSlots = ExpressionUtils.inferNotNullSlots(on,
ctx.cascadesContext);
- if (!conditionSlot.equals(notNullSlots)) {
+ if (conditionSlot.isEmpty() || !conditionSlot.equals(notNullSlots)) {
return null;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 07/19: [Enhancement] [plsql] support drop store procedure (#30966) (#31001)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 98a1b12ac6ddbeecbf1f9b9cd6abdb50c602eede
Author: Vallish Pai <va...@gmail.com>
AuthorDate: Wed Feb 21 20:30:30 2024 +0530
[Enhancement] [plsql] support drop store procedure (#30966) (#31001)
---
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 1 +
.../antlr4/org/apache/doris/nereids/PLParser.g4 | 4 ++
.../doris/nereids/parser/LogicalPlanBuilder.java | 9 ++++
.../apache/doris/nereids/trees/plans/PlanType.java | 3 +-
.../trees/plans/commands/DropProcedureCommand.java | 62 ++++++++++++++++++++++
.../trees/plans/visitor/CommandVisitor.java | 5 ++
.../src/main/java/org/apache/doris/plsql/Exec.java | 33 ++++++++++++
.../suites/plsql_p0/test_plsql_loop_cursor.groovy | 4 ++
8 files changed, 120 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index 0389ac5e090..0def8c10ab3 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -37,6 +37,7 @@ statement
: statementBase # statementBaseAlias
| CALL name=multipartIdentifier LEFT_PAREN (expression (COMMA expression)*)? RIGHT_PAREN #callProcedure
| (ALTER | CREATE (OR REPLACE)? | REPLACE) (PROCEDURE | PROC) name=multipartIdentifier LEFT_PAREN .*? RIGHT_PAREN .*? #createProcedure
+ | DROP (PROCEDURE | PROC) (IF EXISTS)? name=multipartIdentifier #dropProcedure
;
statementBase
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/PLParser.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/PLParser.g4
index e2975f65d00..e49dd9d26b6 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/PLParser.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/PLParser.g4
@@ -98,6 +98,7 @@ stmt :
| create_package_body_stmt
| create_procedure_stmt
| declare_stmt
+ | drop_procedure_stmt
| exec_stmt
| exit_stmt
| fetch_stmt
@@ -329,6 +330,9 @@ package_body_item :
create_procedure_stmt :
(ALTER | CREATE (OR REPLACE)? | REPLACE) (PROCEDURE | PROC) multipartIdentifier create_routine_params? create_routine_options? (AS | IS)? declare_block_inplace? label_stmt? procedure_block (ident_pl SEMICOLON)?
;
+drop_procedure_stmt:
+ DROP (PROCEDURE | PROC) (IF EXISTS)? name=multipartIdentifier
+ ;
create_routine_params :
LEFT_PAREN RIGHT_PAREN
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index d408e446b1f..2a6a602b71d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -83,6 +83,7 @@ import org.apache.doris.nereids.DorisParser.DeleteContext;
import org.apache.doris.nereids.DorisParser.DereferenceContext;
import org.apache.doris.nereids.DorisParser.DropConstraintContext;
import org.apache.doris.nereids.DorisParser.DropMTMVContext;
+import org.apache.doris.nereids.DorisParser.DropProcedureContext;
import org.apache.doris.nereids.DorisParser.ElementAtContext;
import org.apache.doris.nereids.DorisParser.ExistContext;
import org.apache.doris.nereids.DorisParser.ExplainContext;
@@ -348,6 +349,7 @@ import org.apache.doris.nereids.trees.plans.commands.DeleteFromCommand;
import org.apache.doris.nereids.trees.plans.commands.DeleteFromUsingCommand;
import org.apache.doris.nereids.trees.plans.commands.DropConstraintCommand;
import org.apache.doris.nereids.trees.plans.commands.DropMTMVCommand;
+import org.apache.doris.nereids.trees.plans.commands.DropProcedureCommand;
import org.apache.doris.nereids.trees.plans.commands.ExplainCommand;
import org.apache.doris.nereids.trees.plans.commands.ExplainCommand.ExplainLevel;
import org.apache.doris.nereids.trees.plans.commands.ExportCommand;
@@ -3286,4 +3288,11 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor<Object> {
return createProcedurePlan;
});
}
+
+ @Override
+ public LogicalPlan visitDropProcedure(DropProcedureContext ctx) {
+ List<String> nameParts = visitMultipartIdentifier(ctx.name);
+ FuncNameInfo procedureName = new FuncNameInfo(nameParts);
+ return ParserUtils.withOrigin(ctx, () -> new DropProcedureCommand(procedureName, getOriginSql(ctx)));
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java
index e338bb03ffb..e09d8af389d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java
@@ -140,5 +140,6 @@ public enum PlanType {
RESUME_MTMV_COMMAND,
CANCEL_MTMV_TASK_COMMAND,
CALL_COMMAND,
- CREATE_PROCEDURE_COMMAND
+ CREATE_PROCEDURE_COMMAND,
+ DROP_PROCEDURE_COMMAND
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropProcedureCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropProcedureCommand.java
new file mode 100644
index 00000000000..6aa76197d82
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropProcedureCommand.java
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.nereids.annotation.Developing;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.commands.info.FuncNameInfo;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.plsql.metastore.PlsqlMetaClient;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Objects;
+
+/**
+ * Drop table procedure
+ */
+@Developing
+public class DropProcedureCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(DropProcedureCommand.class);
+ private final FuncNameInfo procedureName;
+ private final String source; // Original SQL, from LogicalPlanBuilder.getOriginSql()
+ private final PlsqlMetaClient client;
+
+ /**
+ * constructor
+ */
+ public DropProcedureCommand(FuncNameInfo procedureName, String source) {
+ super(PlanType.DROP_PROCEDURE_COMMAND);
+ this.client = new PlsqlMetaClient();
+ this.procedureName = Objects.requireNonNull(procedureName, "procedureName is null");
+ this.source = Objects.requireNonNull(source, "source is null");
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ ctx.getPlSqlOperation().getExec().functions.remove(procedureName);
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitDropProcedureCommand(this, context);
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/CommandVisitor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/CommandVisitor.java
index 107cfd0a1a2..07e9020f1ff 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/CommandVisitor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/CommandVisitor.java
@@ -31,6 +31,7 @@ import org.apache.doris.nereids.trees.plans.commands.DeleteFromCommand;
import org.apache.doris.nereids.trees.plans.commands.DeleteFromUsingCommand;
import org.apache.doris.nereids.trees.plans.commands.DropConstraintCommand;
import org.apache.doris.nereids.trees.plans.commands.DropMTMVCommand;
+import org.apache.doris.nereids.trees.plans.commands.DropProcedureCommand;
import org.apache.doris.nereids.trees.plans.commands.ExplainCommand;
import org.apache.doris.nereids.trees.plans.commands.ExportCommand;
import org.apache.doris.nereids.trees.plans.commands.InsertIntoTableCommand;
@@ -141,4 +142,8 @@ public interface CommandVisitor<R, C> {
default R visitCreateProcedureCommand(CreateProcedureCommand createProcedureCommand, C context) {
return visitCommand(createProcedureCommand, context);
}
+
+ default R visitDropProcedureCommand(DropProcedureCommand dropProcedureCommand, C context) {
+ return visitCommand(dropProcedureCommand, context);
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/plsql/Exec.java b/fe/fe-core/src/main/java/org/apache/doris/plsql/Exec.java
index 9a042623da3..98ebebfd4ea 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/plsql/Exec.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/plsql/Exec.java
@@ -47,6 +47,7 @@ import org.apache.doris.nereids.PLParser.Declare_cursor_itemContext;
import org.apache.doris.nereids.PLParser.Declare_handler_itemContext;
import org.apache.doris.nereids.PLParser.Declare_var_itemContext;
import org.apache.doris.nereids.PLParser.Doris_statementContext;
+import org.apache.doris.nereids.PLParser.Drop_procedure_stmtContext;
import org.apache.doris.nereids.PLParser.DtypeContext;
import org.apache.doris.nereids.PLParser.Dtype_lenContext;
import org.apache.doris.nereids.PLParser.Exception_block_itemContext;
@@ -1459,6 +1460,38 @@ public class Exec extends org.apache.doris.nereids.PLParserBaseVisitor<Integer>
return null;
}
+ /**
+ * DROP PROCEDURE statement
+ */
+ @Override
+ public Integer visitDrop_procedure_stmt(Drop_procedure_stmtContext ctx) {
+ FuncNameInfo procedureName = new FuncNameInfo(
+ exec.logicalPlanBuilder.visitMultipartIdentifier(ctx.multipartIdentifier()));
+ if (builtinFunctions.exists(procedureName.toString())) {
+ exec.info(ctx, procedureName.toString() + " is a built-in function which cannot be removed.");
+ return 0;
+ }
+ if (trace) {
+ trace(ctx, "DROP PROCEDURE " + procedureName.toString());
+ }
+ exec.functions.remove(procedureName);
+ removeLocalUdf(ctx);
+ return 0;
+ }
+
+ /**
+ * Remove functions and procedures defined in the current script
+ */
+ void removeLocalUdf(ParserRuleContext ctx) {
+ if (exec == this) {
+ String str = Exec.getFormattedText(ctx);
+ int i = localUdf.indexOf(str);
+ if (i != -1) {
+ localUdf.delete(i, i + str.length());
+ }
+ }
+ }
+
@Override
public Integer visitSet_doris_session_option(
Set_doris_session_optionContext ctx) {
diff --git a/regression-test/suites/plsql_p0/test_plsql_loop_cursor.groovy b/regression-test/suites/plsql_p0/test_plsql_loop_cursor.groovy
index 0faa8228fa7..72297888aaa 100644
--- a/regression-test/suites/plsql_p0/test_plsql_loop_cursor.groovy
+++ b/regression-test/suites/plsql_p0/test_plsql_loop_cursor.groovy
@@ -59,4 +59,8 @@ suite("test_plsql_loop_cursor") {
// TODO support print
sql """call procedure_cursor_select(111, "plsql111")"""
sql """call procedure_cursor_select(111, "plsql333")"""
+ // TODO call show command before drop
+ sql """DROP PROCEDURE procedure_cursor_select"""
+ sql """DROP PROC procedure_insert"""
+ // TODO call show command after drop
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 19/19: [fix](txn insert) Txn insert can not write to table with mv (#31167)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ef22bd3318c8158455aab4b6cad2f215d30dc1be
Author: meiyi <my...@gmail.com>
AuthorDate: Thu Feb 22 12:15:19 2024 +0800
[fix](txn insert) Txn insert can not write to table with mv (#31167)
---
.../apache/doris/analysis/NativeInsertStmt.java | 54 +++++++++++-----------
regression-test/data/insert_p0/txn_insert.out | 12 +++++
regression-test/suites/insert_p0/txn_insert.groovy | 21 ++++++++-
3 files changed, 60 insertions(+), 27 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java
index 2cdec74cd9a..1b0d8e59dd2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java
@@ -550,7 +550,6 @@ public class NativeInsertStmt extends InsertStmt {
private void analyzeSubquery(Analyzer analyzer, boolean skipCheck) throws UserException {
// Analyze columns mentioned in the statement.
Set<String> mentionedColumns = Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER);
- List<String> realTargetColumnNames;
if (targetColumnNames == null) {
hasEmptyTargetColumns = true;
// the mentioned columns are columns which are visible to user, so here we use
@@ -600,35 +599,38 @@ public class NativeInsertStmt extends InsertStmt {
* will be used in as a mapping from queryStmt.getResultExprs() to targetColumns define expr
*/
List<Pair<Integer, Column>> origColIdxsForExtendCols = Lists.newArrayList();
- for (Column column : targetTable.getFullSchema()) {
- if (column.isNameWithPrefix(SchemaChangeHandler.SHADOW_NAME_PREFIX)) {
- String origName = Column.removeNamePrefix(column.getName());
- for (int i = 0; i < targetColumns.size(); i++) {
- if (targetColumns.get(i).nameEquals(origName, false)) {
- // Rule A
- origColIdxsForExtendCols.add(Pair.of(i, null));
- targetColumns.add(column);
- break;
- }
- }
- }
- if (column.isNameWithPrefix(CreateMaterializedViewStmt.MATERIALIZED_VIEW_NAME_PREFIX)
- || column.isNameWithPrefix(CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX)) {
- List<SlotRef> refColumns = column.getRefColumns();
- if (refColumns == null) {
- ErrorReport.reportAnalysisException(ErrorCode.ERR_BAD_FIELD_ERROR,
- column.getName(), targetTable.getName());
- }
- for (SlotRef refColumn : refColumns) {
- String origName = refColumn.getColumnName();
- for (int originColumnIdx = 0; originColumnIdx < targetColumns.size(); originColumnIdx++) {
- if (targetColumns.get(originColumnIdx).nameEquals(origName, false)) {
- origColIdxsForExtendCols.add(Pair.of(originColumnIdx, column));
+ if (!ConnectContext.get().isTxnModel()) {
+ for (Column column : targetTable.getFullSchema()) {
+ if (column.isNameWithPrefix(SchemaChangeHandler.SHADOW_NAME_PREFIX)) {
+ String origName = Column.removeNamePrefix(column.getName());
+ for (int i = 0; i < targetColumns.size(); i++) {
+ if (targetColumns.get(i).nameEquals(origName, false)) {
+ // Rule A
+ origColIdxsForExtendCols.add(Pair.of(i, null));
targetColumns.add(column);
break;
}
}
}
+ if (column.isNameWithPrefix(CreateMaterializedViewStmt.MATERIALIZED_VIEW_NAME_PREFIX)
+ || column.isNameWithPrefix(
+ CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX)) {
+ List<SlotRef> refColumns = column.getRefColumns();
+ if (refColumns == null) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_BAD_FIELD_ERROR,
+ column.getName(), targetTable.getName());
+ }
+ for (SlotRef refColumn : refColumns) {
+ String origName = refColumn.getColumnName();
+ for (int originColumnIdx = 0; originColumnIdx < targetColumns.size(); originColumnIdx++) {
+ if (targetColumns.get(originColumnIdx).nameEquals(origName, false)) {
+ origColIdxsForExtendCols.add(Pair.of(originColumnIdx, column));
+ targetColumns.add(column);
+ break;
+ }
+ }
+ }
+ }
}
}
@@ -669,7 +671,7 @@ public class NativeInsertStmt extends InsertStmt {
checkColumnCoverage(mentionedColumns, targetTable.getBaseSchema());
}
- realTargetColumnNames = targetColumns.stream().map(Column::getName).collect(Collectors.toList());
+ List<String> realTargetColumnNames = targetColumns.stream().map(Column::getName).collect(Collectors.toList());
// handle VALUES() or SELECT constant list
if (isValuesOrConstantSelect) {
diff --git a/regression-test/data/insert_p0/txn_insert.out b/regression-test/data/insert_p0/txn_insert.out
index 41992dcd3c1..f4947a2ac65 100644
--- a/regression-test/data/insert_p0/txn_insert.out
+++ b/regression-test/data/insert_p0/txn_insert.out
@@ -27,3 +27,15 @@
2 3.3 xyz [1] [1, 0]
2 3.3 xyz [1] [1, 0]
+-- !select5 --
+1 2
+3 4
+5 6
+7 8
+
+-- !select6 --
+2
+4
+6
+8
+
diff --git a/regression-test/suites/insert_p0/txn_insert.groovy b/regression-test/suites/insert_p0/txn_insert.groovy
index 64477ee8c3e..6e745073b1b 100644
--- a/regression-test/suites/insert_p0/txn_insert.groovy
+++ b/regression-test/suites/insert_p0/txn_insert.groovy
@@ -1,4 +1,3 @@
-
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
@@ -72,4 +71,24 @@ suite("txn_insert") {
sql "rollback"
sql "sync"
order_qt_select4 """select * from $table"""
+
+ // write to table with mv
+ table = table + "_mv"
+ sql """ DROP TABLE IF EXISTS $table """
+ sql """
+ create table $table (
+ id int default '10',
+ c1 int default '10'
+ ) distributed by hash(id, c1)
+ properties('replication_num'="1");
+ """
+ createMV """ create materialized view mv_${table} as select c1 from $table; """
+ sql "begin"
+ sql """insert into $table values(1, 2), (3, 4)"""
+ sql """insert into $table values(5, 6)"""
+ sql """insert into $table values(7, 8)"""
+ sql "commit"
+ sql "sync"
+ order_qt_select5 """select * from $table"""
+ order_qt_select6 """select c1 from $table"""
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 16/19: [bugfix](docs)Fix some spelling mistakes in english doc (#31241)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6096227fea3d6801d7da2191bce6bca389df65be
Author: Vallish Pai <va...@gmail.com>
AuthorDate: Thu Feb 22 07:33:01 2024 +0530
[bugfix](docs)Fix some spelling mistakes in english doc (#31241)
---
docs/en/community/design/doris_storage_optimization.md | 4 ++--
docs/en/community/developer-guide/cpp-diagnostic-code.md | 2 +-
docs/en/community/how-to-contribute/commit-format-specification.md | 2 +-
docs/en/community/join-community.md | 2 +-
docs/en/community/release-and-verify/release-complete.md | 2 +-
docs/en/docs/admin-manual/http-actions/fe/debug-point-action.md | 6 +++---
docs/en/docs/admin-manual/http-actions/fe/metrics-action.md | 2 +-
docs/en/docs/admin-manual/privilege-ldap/ranger.md | 2 +-
docs/en/docs/advanced/compute-node.md | 2 +-
docs/en/docs/advanced/partition/auto-partition.md | 2 +-
10 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/docs/en/community/design/doris_storage_optimization.md b/docs/en/community/design/doris_storage_optimization.md
index 3172e0b60b0..2ebfb680cfd 100644
--- a/docs/en/community/design/doris_storage_optimization.md
+++ b/docs/en/community/design/doris_storage_optimization.md
@@ -125,8 +125,8 @@ message ColumnPB {
required string type = 3; // Column type
optional bool is_key = 4; // Whether column is a primary key column
optional string aggregation = 5; // Aggregate type
- optional bool is_nullable = 6; // Whether column is allowed to assgin null
- optional bytes default_value = 7; // Defalut value
+ optional bool is_nullable = 6; // Whether column is allowed to assign null
+ optional bytes default_value = 7; // Default value
optional int32 precision = 8; // Precision of column
optional int32 frac = 9;
optional int32 length = 10; // Length of column
diff --git a/docs/en/community/developer-guide/cpp-diagnostic-code.md b/docs/en/community/developer-guide/cpp-diagnostic-code.md
index ebfde51503c..13cdbd15f4b 100644
--- a/docs/en/community/developer-guide/cpp-diagnostic-code.md
+++ b/docs/en/community/developer-guide/cpp-diagnostic-code.md
@@ -30,7 +30,7 @@ Doris support to use [Clangd](https://clangd.llvm.org/) and [Clang-Tidy](https:/
to diagnostic code. Clangd and Clang-Tidy already has in [LDB-toolchain](/docs/install/source-install/compilation-with-ldb-toolchain)，also can install by self.
### Clang-Tidy
-Clang-Tidy can do some diagnostic cofig, config file `.clang-tidy` is in Doris root path. Compared with vscode-cpptools, clangd can provide more powerful and accurate code jumping for vscode, and integrates the analysis and quick-fix functions of clang-tidy.
+Clang-Tidy can do some diagnostic config, config file `.clang-tidy` is in Doris root path. Compared with vscode-cpptools, clangd can provide more powerful and accurate code jumping for vscode, and integrates the analysis and quick-fix functions of clang-tidy.
### Enable clangd on VSCODE
diff --git a/docs/en/community/how-to-contribute/commit-format-specification.md b/docs/en/community/how-to-contribute/commit-format-specification.md
index b42b9a17f8b..e466ef00ac8 100644
--- a/docs/en/community/how-to-contribute/commit-format-specification.md
+++ b/docs/en/community/how-to-contribute/commit-format-specification.md
@@ -41,7 +41,7 @@ Commit is divided into ‘ title ’ and ‘ content ’ , the title should be l
* fix: Bug fix
* feature: New feature
- * feature-wip: Feature works-in-porgress.
+ * feature-wip: Feature works-in-progress.
* improvement: Optimization and improvement for the original feature.
* style: Code style adjustment
* typo: Code or Document correction
diff --git a/docs/en/community/join-community.md b/docs/en/community/join-community.md
index 984ca2720f8..b50518e1f45 100644
--- a/docs/en/community/join-community.md
+++ b/docs/en/community/join-community.md
@@ -27,7 +27,7 @@ under the License.
**Thrive together in [Apache Doris](https://github.com/apache/doris) community with users and developers from all around the world.**
-We have graduated from Apache incubator succesffully and become an Top-Level Project in June 2022.
+We have graduated from Apache incubator successfully and become an Top-Level Project in June 2022.
<hr />
diff --git a/docs/en/community/release-and-verify/release-complete.md b/docs/en/community/release-and-verify/release-complete.md
index ea0ec10604c..6790c2ffff5 100644
--- a/docs/en/community/release-and-verify/release-complete.md
+++ b/docs/en/community/release-and-verify/release-complete.md
@@ -78,7 +78,7 @@ https://www.apache.org/dist/doris/1.xx/
http://www.apache.org/dyn/closer.cgi/doris/1.xx/apache-doris-1.xx-src.tar.gz
```
-Ssource package:
+Source package:
```
http://www.apache.org/dyn/closer.cgi/doris/1.xx/apache-doris-1.xx-src.tar.gz
diff --git a/docs/en/docs/admin-manual/http-actions/fe/debug-point-action.md b/docs/en/docs/admin-manual/http-actions/fe/debug-point-action.md
index f28ba12b9b2..16313694732 100644
--- a/docs/en/docs/admin-manual/http-actions/fe/debug-point-action.md
+++ b/docs/en/docs/admin-manual/http-actions/fe/debug-point-action.md
@@ -129,7 +129,7 @@ curl -X POST "http://127.0.0.1:8030/api/debug_point/add/foo?execute=5"
## Pass Custom Parameters
When activating debug point, besides "timeout" and "execute" mentioned above, passing custom parameters is also allowed.<br/>
-A parameter is a key-value pair in the form of "key=value" in url path, after debug point name glued by charactor '?'.<br/>
+A parameter is a key-value pair in the form of "key=value" in url path, after debug point name glued by character '?'.<br/>
See examples below.
### API
@@ -296,8 +296,8 @@ and `disableDebugPointForAllFEs()` or `disableDebugPointForAllBEs()` needs to be
### Concurrent Issue
-Enabled debug points affects FE or BE globally, which could cause other concurrent tests to fail unexpectly in your pull request. <br/>
-To avoid this, there's a convension that regression tests using debug points must be in directory regression-test/suites/fault_injection_p0, <br/>
+Enabled debug points affects FE or BE globally, which could cause other concurrent tests to fail unexpectedly in your pull request. <br/>
+To avoid this, there's a convention that regression tests using debug points must be in directory regression-test/suites/fault_injection_p0, <br/>
and their group name must be "nonConcurrent", as these regression tests will be executed serially by pull request workflow.
### Examples
diff --git a/docs/en/docs/admin-manual/http-actions/fe/metrics-action.md b/docs/en/docs/admin-manual/http-actions/fe/metrics-action.md
index 0d7c3f0e669..2168c4152cc 100644
--- a/docs/en/docs/admin-manual/http-actions/fe/metrics-action.md
+++ b/docs/en/docs/admin-manual/http-actions/fe/metrics-action.md
@@ -32,7 +32,7 @@ under the License.
## Description
-Used to obtain doris metrics infomation.
+Used to obtain doris metrics information.
## Path parameters
diff --git a/docs/en/docs/admin-manual/privilege-ldap/ranger.md b/docs/en/docs/admin-manual/privilege-ldap/ranger.md
index c2778aafa21..06265e3a8e6 100644
--- a/docs/en/docs/admin-manual/privilege-ldap/ranger.md
+++ b/docs/en/docs/admin-manual/privilege-ldap/ranger.md
@@ -122,7 +122,7 @@ Click the `+` button next to the plugin to add a Doris service:
The meaning of some parameters of Config Properties is as follows:
-- `Username`/`Pasword`: the username and password of the Doris cluster. It is recommended to use the Admin user here.
+- `Username`/`Password`: the username and password of the Doris cluster. It is recommended to use the Admin user here.
- `jdbc.driver_class`: Connect to the JDBC driver used by Doris. `com.mysql.cj.jdbc.Driver`
- `jdbc.url`: JDBC url connection string of Doris cluster. `jdbc:mysql://172.21.0.101:9030?useSSL=false`
- Additional parameters:
diff --git a/docs/en/docs/advanced/compute-node.md b/docs/en/docs/advanced/compute-node.md
index 2dbc2e9290b..ec31052aaaa 100644
--- a/docs/en/docs/advanced/compute-node.md
+++ b/docs/en/docs/advanced/compute-node.md
@@ -62,7 +62,7 @@ Add configuration items to BE's configuration file `be.conf`:
be_node_role = computation
```
-This defualt value of this is `mix`, and this is original BE node type. After setting to `computation`, the node is a computing node.
+This default value of this is `mix`, and this is original BE node type. After setting to `computation`, the node is a computing node.
You can see the value of the'NodeRole 'field through the `show backends\G` command. If it is'mix ', it is a mixed node, and if it is'computation', it is a computing node
diff --git a/docs/en/docs/advanced/partition/auto-partition.md b/docs/en/docs/advanced/partition/auto-partition.md
index 1f9e8fa1e97..cae9bcb2669 100644
--- a/docs/en/docs/advanced/partition/auto-partition.md
+++ b/docs/en/docs/advanced/partition/auto-partition.md
@@ -36,7 +36,7 @@ The Auto Partitioning feature supports automatic detection of whether the corres
The auto partition function mainly solves the problem that the user expects to partition the table based on a certain column, but the data distribution of the column is scattered or unpredictable, so it is difficult to accurately create the required partitions when building or adjusting the structure of the table, or the number of partitions is so large that it is too cumbersome to create them manually.
-Take the time type partition column as an example, in the [Dynamic Partition](./dynamic-partition) function, we support the automatic creation of new partitions to accommodate real-time data at specific time periods. For real-time user behaviour logs and other scenarios, this feature basically meets the requirements. However, in more complex scenarios, such as dealing with non-real-time data, the partition column is independent of the current system time and contains a large number of di [...]
+Take the time type partition column as an example, in the [Dynamic Partition](./dynamic-partition) function, we support the automatic creation of new partitions to accommodate real-time data at specific time periods. For real-time user behavior logs and other scenarios, this feature basically meets the requirements. However, in more complex scenarios, such as dealing with non-real-time data, the partition column is independent of the current system time and contains a large number of dis [...]
Suppose our table DDL is as follows:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 06/19: [Exec](RF) Support merge remote rf local first (#31067)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c56cb0ac3e5094f762bf13e3b83964c73391a524
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Wed Feb 21 22:38:33 2024 +0800
[Exec](RF) Support merge remote rf local first (#31067)
---
be/src/agent/be_exec_version_manager.h | 1 +
be/src/exprs/runtime_filter.cpp | 77 +++++++++---------
be/src/exprs/runtime_filter.h | 24 ++----
be/src/exprs/runtime_filter_slots.h | 11 ++-
be/src/pipeline/exec/datagen_operator.cpp | 9 +--
be/src/pipeline/exec/hashjoin_build_sink.cpp | 23 +++---
be/src/pipeline/exec/hashjoin_build_sink.h | 2 +-
.../exec/nested_loop_join_build_operator.cpp | 4 +-
be/src/pipeline/pipeline_fragment_context.cpp | 5 +-
be/src/pipeline/pipeline_fragment_context.h | 2 +-
.../pipeline_x/pipeline_x_fragment_context.cpp | 14 ++--
.../pipeline_x/pipeline_x_fragment_context.h | 2 +-
be/src/runtime/fragment_mgr.cpp | 5 +-
be/src/runtime/query_context.h | 3 +
be/src/runtime/runtime_filter_mgr.cpp | 91 ++++++++++++++++++----
be/src/runtime/runtime_filter_mgr.h | 32 ++++----
be/src/runtime/runtime_state.cpp | 36 +++++++--
be/src/runtime/runtime_state.h | 22 ++++--
be/src/vec/exec/join/vhash_join_node.cpp | 6 +-
be/src/vec/exec/join/vnested_loop_join_node.cpp | 4 +-
be/src/vec/exec/runtime_filter_consumer.cpp | 23 ++----
be/src/vec/exec/runtime_filter_consumer.h | 4 +-
be/src/vec/exec/vdata_gen_scan_node.cpp | 9 +--
.../main/java/org/apache/doris/qe/Coordinator.java | 3 +-
24 files changed, 243 insertions(+), 169 deletions(-)
diff --git a/be/src/agent/be_exec_version_manager.h b/be/src/agent/be_exec_version_manager.h
index f5213c54089..afe738684aa 100644
--- a/be/src/agent/be_exec_version_manager.h
+++ b/be/src/agent/be_exec_version_manager.h
@@ -65,6 +65,7 @@ private:
* d. unix_timestamp function support timestamp with float for datetimev2, and change nullable mode.
* e. change shuffle serialize/deserialize way
* f. shrink some function's nullable mode.
+ * g. do local merge of remote runtime filter
*/
constexpr inline int BeExecVersionManager::max_be_exec_version = 3;
constexpr inline int BeExecVersionManager::min_be_exec_version = 0;
diff --git a/be/src/exprs/runtime_filter.cpp b/be/src/exprs/runtime_filter.cpp
index 2ac9ad40c5a..1ec66bf2a87 100644
--- a/be/src/exprs/runtime_filter.cpp
+++ b/be/src/exprs/runtime_filter.cpp
@@ -938,11 +938,11 @@ private:
Status IRuntimeFilter::create(RuntimeFilterParamsContext* state, ObjectPool* pool,
const TRuntimeFilterDesc* desc, const TQueryOptions* query_options,
const RuntimeFilterRole role, int node_id, IRuntimeFilter** res,
- bool build_bf_exactly, bool is_global, int parallel_tasks) {
- *res = pool->add(new IRuntimeFilter(state, pool, desc, is_global, parallel_tasks));
+ bool build_bf_exactly, bool need_local_merge) {
+ *res = pool->add(new IRuntimeFilter(state, pool, desc, need_local_merge));
(*res)->set_role(role);
return (*res)->init_with_desc(desc, query_options, node_id,
- is_global ? false : build_bf_exactly);
+ need_local_merge ? false : build_bf_exactly);
}
vectorized::SharedRuntimeFilterContext& IRuntimeFilter::get_shared_context_ref() {
@@ -954,47 +954,53 @@ void IRuntimeFilter::insert_batch(const vectorized::ColumnPtr column, size_t sta
_wrapper->insert_batch(column, start);
}
-Status IRuntimeFilter::merge_local_filter(RuntimePredicateWrapper* wrapper, int* merged_num) {
- SCOPED_TIMER(_merge_local_rf_timer);
- std::unique_lock lock(_local_merge_mutex);
- if (_merged_rf_num == 0) {
- _wrapper = wrapper;
- } else {
- RETURN_IF_ERROR(merge_from(wrapper));
- }
- *merged_num = ++_merged_rf_num;
- return Status::OK();
-}
-
Status IRuntimeFilter::publish(bool publish_local) {
DCHECK(is_producer());
- if (_is_global && _has_local_target) {
- std::vector<IRuntimeFilter*> filters;
- RETURN_IF_ERROR(_state->get_query_ctx()->runtime_filter_mgr()->get_consume_filters(
- _filter_id, filters));
- // push down
- for (auto filter : filters) {
- int merged_num = 0;
- RETURN_IF_ERROR(filter->merge_local_filter(_wrapper, &merged_num));
- if (merged_num == _parallel_build_tasks) {
- filter->update_runtime_filter_type_to_profile();
- filter->signal();
- }
- }
- } else if (_has_local_target) {
+ auto send_to_remote = [&](IRuntimeFilter* filter) {
+ TNetworkAddress addr;
+ DCHECK(_state != nullptr);
+ RETURN_IF_ERROR(_state->runtime_filter_mgr->get_merge_addr(&addr));
+ return filter->push_to_remote(&addr, _opt_remote_rf);
+ };
+ auto send_to_local = [&](RuntimePredicateWrapper* wrapper) {
std::vector<IRuntimeFilter*> filters;
RETURN_IF_ERROR(_state->runtime_filter_mgr->get_consume_filters(_filter_id, filters));
+ DCHECK(!filters.empty());
// push down
for (auto filter : filters) {
- filter->_wrapper = _wrapper;
+ filter->_wrapper = wrapper;
filter->update_runtime_filter_type_to_profile();
filter->signal();
}
+ return Status::OK();
+ };
+ auto do_local_merge = [&]() {
+ LocalMergeFilters* local_merge_filters = nullptr;
+ RETURN_IF_ERROR(_state->runtime_filter_mgr->get_local_merge_producer_filters(
+ _filter_id, &local_merge_filters));
+ std::lock_guard l(*local_merge_filters->lock);
+ RETURN_IF_ERROR(local_merge_filters->filters[0]->merge_from(_wrapper));
+ local_merge_filters->merge_time--;
+ if (local_merge_filters->merge_time == 0) {
+ if (_has_local_target) {
+ RETURN_IF_ERROR(send_to_local(local_merge_filters->filters[0]->_wrapper));
+ } else {
+ RETURN_IF_ERROR(send_to_remote(local_merge_filters->filters[0]));
+ }
+ }
+ return Status::OK();
+ };
+
+ if (_need_local_merge && _has_local_target) {
+ RETURN_IF_ERROR(do_local_merge());
+ } else if (_has_local_target) {
+ RETURN_IF_ERROR(send_to_local(_wrapper));
} else if (!publish_local) {
- TNetworkAddress addr;
- DCHECK(_state != nullptr);
- RETURN_IF_ERROR(_state->runtime_filter_mgr->get_merge_addr(&addr));
- return push_to_remote(&addr, _opt_remote_rf);
+ if (_is_broadcast_join || _state->be_exec_version < 3) {
+ RETURN_IF_ERROR(send_to_remote(this));
+ } else {
+ RETURN_IF_ERROR(do_local_merge());
+ }
} else {
// remote broadcast join only push onetime in build shared hash table
// publish_local only set true on copy shared hash table
@@ -1366,9 +1372,6 @@ void IRuntimeFilter::init_profile(RuntimeProfile* parent_profile) {
_profile_init = true;
parent_profile->add_child(_profile.get(), true, nullptr);
_profile->add_info_string("Info", _format_status());
- if (_is_global) {
- _merge_local_rf_timer = ADD_TIMER(_profile.get(), "MergeLocalRuntimeFilterTime");
- }
if (_runtime_filter_type == RuntimeFilterType::IN_OR_BLOOM_FILTER) {
update_runtime_filter_type_to_profile();
}
diff --git a/be/src/exprs/runtime_filter.h b/be/src/exprs/runtime_filter.h
index ae8063e5610..d853493889c 100644
--- a/be/src/exprs/runtime_filter.h
+++ b/be/src/exprs/runtime_filter.h
@@ -185,7 +185,7 @@ enum RuntimeFilterState {
class IRuntimeFilter {
public:
IRuntimeFilter(RuntimeFilterParamsContext* state, ObjectPool* pool,
- const TRuntimeFilterDesc* desc, bool is_global = false, int parallel_tasks = -1)
+ const TRuntimeFilterDesc* desc, bool need_local_merge = false)
: _state(state),
_pool(pool),
_filter_id(desc->filter_id),
@@ -204,16 +204,14 @@ public:
_name(fmt::format("RuntimeFilter: (id = {}, type = {})", _filter_id,
to_string(_runtime_filter_type))),
_profile(new RuntimeProfile(_name)),
- _is_global(is_global),
- _parallel_build_tasks(parallel_tasks) {}
+ _need_local_merge(need_local_merge) {}
~IRuntimeFilter() = default;
static Status create(RuntimeFilterParamsContext* state, ObjectPool* pool,
const TRuntimeFilterDesc* desc, const TQueryOptions* query_options,
const RuntimeFilterRole role, int node_id, IRuntimeFilter** res,
- bool build_bf_exactly = false, bool is_global = false,
- int parallel_tasks = 0);
+ bool build_bf_exactly = false, bool need_local_merge = false);
vectorized::SharedRuntimeFilterContext& get_shared_context_ref();
@@ -349,8 +347,6 @@ public:
void set_filter_timer(std::shared_ptr<pipeline::RuntimeFilterTimer>);
- Status merge_local_filter(RuntimePredicateWrapper* wrapper, int* merged_num);
-
protected:
// serialize _wrapper to protobuf
void to_protobuf(PInFilter* filter);
@@ -426,18 +422,10 @@ protected:
// parent profile
// only effect on consumer
std::unique_ptr<RuntimeProfile> _profile;
- RuntimeProfile::Counter* _merge_local_rf_timer = nullptr;
bool _opt_remote_rf;
- // `_is_global` indicates whether this runtime filter is global on this BE.
- // All runtime filters should be merged on each BE if it is global.
- // This is improvement for pipelineX.
- const bool _is_global = false;
- std::mutex _local_merge_mutex;
- // There are `_parallel_build_tasks` pipeline tasks to build runtime filter.
- // We should call `signal` once all runtime filters are done and merged to one
- // (e.g. `_merged_rf_num` is equal to `_parallel_build_tasks`).
- int _merged_rf_num = 0;
- const int _parallel_build_tasks = -1;
+ // `_need_local_merge` indicates whether this runtime filter is global on this BE.
+ // All runtime filters should be merged on each BE before push_to_remote or publish.
+ const bool _need_local_merge = false;
std::vector<std::shared_ptr<pipeline::RuntimeFilterTimer>> _filter_timer;
};
diff --git a/be/src/exprs/runtime_filter_slots.h b/be/src/exprs/runtime_filter_slots.h
index 7a738b8c06d..7f34bf7f2c9 100644
--- a/be/src/exprs/runtime_filter_slots.h
+++ b/be/src/exprs/runtime_filter_slots.h
@@ -34,25 +34,24 @@ class VRuntimeFilterSlots {
public:
VRuntimeFilterSlots(
const std::vector<std::shared_ptr<vectorized::VExprContext>>& build_expr_ctxs,
- const std::vector<IRuntimeFilter*>& runtime_filters, bool is_global = false)
+ const std::vector<IRuntimeFilter*>& runtime_filters, bool need_local_merge = false)
: _build_expr_context(build_expr_ctxs),
_runtime_filters(runtime_filters),
- _is_global(is_global) {}
+ _need_local_merge(need_local_merge) {}
Status init(RuntimeState* state, int64_t hash_table_size) {
// runtime filter effect strategy
// 1. we will ignore IN filter when hash_table_size is too big
// 2. we will ignore BLOOM filter and MinMax filter when hash_table_size
// is too small and IN filter has effect
-
std::map<int, bool> has_in_filter;
auto ignore_local_filter = [&](int filter_id) {
// Now pipeline x have bug in ignore, after fix the problem enable ignore logic in pipeline x
- if (_is_global) {
+ if (_need_local_merge) {
return Status::OK();
}
- auto runtime_filter_mgr = state->runtime_filter_mgr();
+ auto runtime_filter_mgr = state->local_runtime_filter_mgr();
std::vector<IRuntimeFilter*> filters;
RETURN_IF_ERROR(runtime_filter_mgr->get_consume_filters(filter_id, filters));
@@ -217,7 +216,7 @@ public:
private:
const std::vector<std::shared_ptr<vectorized::VExprContext>>& _build_expr_context;
std::vector<IRuntimeFilter*> _runtime_filters;
- const bool _is_global = false;
+ const bool _need_local_merge = false;
// prob_contition index -> [IRuntimeFilter]
std::map<int, std::list<IRuntimeFilter*>> _runtime_filters_map;
};
diff --git a/be/src/pipeline/exec/datagen_operator.cpp b/be/src/pipeline/exec/datagen_operator.cpp
index 916ce62aa26..418068ef9d8 100644
--- a/be/src/pipeline/exec/datagen_operator.cpp
+++ b/be/src/pipeline/exec/datagen_operator.cpp
@@ -100,13 +100,8 @@ Status DataGenLocalState::init(RuntimeState* state, LocalStateInfo& info) {
// TODO: use runtime filter to filte result block, maybe this node need derive from vscan_node.
for (const auto& filter_desc : p._runtime_filter_descs) {
IRuntimeFilter* runtime_filter = nullptr;
- if (filter_desc.__isset.opt_remote_rf && filter_desc.opt_remote_rf) {
- RETURN_IF_ERROR(state->get_query_ctx()->runtime_filter_mgr()->register_consumer_filter(
- filter_desc, state->query_options(), p.node_id(), &runtime_filter, false));
- } else {
- RETURN_IF_ERROR(state->runtime_filter_mgr()->register_consumer_filter(
- filter_desc, state->query_options(), p.node_id(), &runtime_filter, false));
- }
+ RETURN_IF_ERROR(state->register_consumer_runtime_filter(filter_desc, false, p.node_id(),
+ &runtime_filter));
runtime_filter->init_profile(_runtime_profile.get());
}
return Status::OK();
diff --git a/be/src/pipeline/exec/hashjoin_build_sink.cpp b/be/src/pipeline/exec/hashjoin_build_sink.cpp
index 19de8fb7bdc..9c6eff4cda8 100644
--- a/be/src/pipeline/exec/hashjoin_build_sink.cpp
+++ b/be/src/pipeline/exec/hashjoin_build_sink.cpp
@@ -99,9 +99,9 @@ Status HashJoinBuildSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo
_hash_table_init(state);
_runtime_filters.resize(p._runtime_filter_descs.size());
for (size_t i = 0; i < p._runtime_filter_descs.size(); i++) {
- RETURN_IF_ERROR(state->runtime_filter_mgr()->register_producer_filter(
- p._runtime_filter_descs[i], state->query_options(), &_runtime_filters[i],
- _build_expr_ctxs.size() == 1, p._use_global_rf, p._child_x->parallel_tasks()));
+ RETURN_IF_ERROR(state->register_producer_runtime_filter(
+ p._runtime_filter_descs[i], p._need_local_merge, &_runtime_filters[i],
+ _build_expr_ctxs.size() == 1));
}
return Status::OK();
@@ -370,7 +370,7 @@ void HashJoinBuildSinkLocalState::_hash_table_init(RuntimeState* state) {
HashJoinBuildSinkOperatorX::HashJoinBuildSinkOperatorX(ObjectPool* pool, int operator_id,
const TPlanNode& tnode,
const DescriptorTbl& descs,
- bool use_global_rf)
+ bool need_local_merge)
: JoinBuildSinkOperatorX(pool, operator_id, tnode, descs),
_join_distribution(tnode.hash_join_node.__isset.dist_type ? tnode.hash_join_node.dist_type
: TJoinDistributionType::NONE),
@@ -379,7 +379,7 @@ HashJoinBuildSinkOperatorX::HashJoinBuildSinkOperatorX(ObjectPool* pool, int ope
_partition_exprs(tnode.__isset.distribute_expr_lists && !_is_broadcast_join
? tnode.distribute_expr_lists[1]
: std::vector<TExpr> {}),
- _use_global_rf(use_global_rf) {}
+ _need_local_merge(need_local_merge) {}
Status HashJoinBuildSinkOperatorX::prepare(RuntimeState* state) {
if (_is_broadcast_join) {
@@ -475,10 +475,11 @@ Status HashJoinBuildSinkOperatorX::sink(RuntimeState* state, vectorized::Block*
local_state._shared_state->build_block = std::make_shared<vectorized::Block>(
local_state._build_side_mutable_block.to_block());
- const bool use_global_rf =
- local_state._parent->cast<HashJoinBuildSinkOperatorX>()._use_global_rf;
+ const bool need_local_merge =
+ local_state._parent->cast<HashJoinBuildSinkOperatorX>()._need_local_merge;
RETURN_IF_ERROR(vectorized::process_runtime_filter_build(
- state, local_state._shared_state->build_block.get(), &local_state, use_global_rf));
+ state, local_state._shared_state->build_block.get(), &local_state,
+ need_local_merge));
RETURN_IF_ERROR(
local_state.process_build_block(state, (*local_state._shared_state->build_block)));
if (_shared_hashtable_controller) {
@@ -528,8 +529,8 @@ Status HashJoinBuildSinkOperatorX::sink(RuntimeState* state, vectorized::Block*
local_state._shared_state->build_block = _shared_hash_table_context->block;
local_state._shared_state->build_indexes_null =
_shared_hash_table_context->build_indexes_null;
- const bool use_global_rf =
- local_state._parent->cast<HashJoinBuildSinkOperatorX>()._use_global_rf;
+ const bool need_local_merge =
+ local_state._parent->cast<HashJoinBuildSinkOperatorX>()._need_local_merge;
if (!_shared_hash_table_context->runtime_filters.empty()) {
auto ret = std::visit(
@@ -545,7 +546,7 @@ Status HashJoinBuildSinkOperatorX::sink(RuntimeState* state, vectorized::Block*
local_state._runtime_filter_slots =
std::make_shared<VRuntimeFilterSlots>(
_build_expr_ctxs, local_state._runtime_filters,
- use_global_rf);
+ need_local_merge);
RETURN_IF_ERROR(local_state._runtime_filter_slots->init(
state, arg.hash_table->size()));
diff --git a/be/src/pipeline/exec/hashjoin_build_sink.h b/be/src/pipeline/exec/hashjoin_build_sink.h
index 56a651e4210..134aba69a48 100644
--- a/be/src/pipeline/exec/hashjoin_build_sink.h
+++ b/be/src/pipeline/exec/hashjoin_build_sink.h
@@ -190,7 +190,7 @@ private:
vectorized::SharedHashTableContextPtr _shared_hash_table_context = nullptr;
const std::vector<TExpr> _partition_exprs;
- const bool _use_global_rf;
+ const bool _need_local_merge;
};
} // namespace pipeline
diff --git a/be/src/pipeline/exec/nested_loop_join_build_operator.cpp b/be/src/pipeline/exec/nested_loop_join_build_operator.cpp
index c30bb5ad67c..8485e2c0b24 100644
--- a/be/src/pipeline/exec/nested_loop_join_build_operator.cpp
+++ b/be/src/pipeline/exec/nested_loop_join_build_operator.cpp
@@ -42,8 +42,8 @@ Status NestedLoopJoinBuildSinkLocalState::init(RuntimeState* state, LocalSinkSta
}
_runtime_filters.resize(p._runtime_filter_descs.size());
for (size_t i = 0; i < p._runtime_filter_descs.size(); i++) {
- RETURN_IF_ERROR(state->runtime_filter_mgr()->register_producer_filter(
- p._runtime_filter_descs[i], state->query_options(), &_runtime_filters[i]));
+ RETURN_IF_ERROR(state->register_producer_runtime_filter(p._runtime_filter_descs[i], false,
+ &_runtime_filters[i], false));
}
return Status::OK();
}
diff --git a/be/src/pipeline/pipeline_fragment_context.cpp b/be/src/pipeline/pipeline_fragment_context.cpp
index e251b203794..358086e94eb 100644
--- a/be/src/pipeline/pipeline_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_fragment_context.cpp
@@ -230,8 +230,9 @@ Status PipelineFragmentContext::prepare(const doris::TPipelineFragmentParams& re
_runtime_state = RuntimeState::create_unique(
local_params.fragment_instance_id, request.query_id, request.fragment_id,
request.query_options, _query_ctx->query_globals, _exec_env, _query_ctx.get());
- if (local_params.__isset.runtime_filter_params) {
- _runtime_state->set_runtime_filter_params(local_params.runtime_filter_params);
+ if (idx == 0 && local_params.__isset.runtime_filter_params) {
+ _query_ctx->runtime_filter_mgr()->set_runtime_filter_params(
+ local_params.runtime_filter_params);
}
_runtime_state->set_task_execution_context(shared_from_this());
diff --git a/be/src/pipeline/pipeline_fragment_context.h b/be/src/pipeline/pipeline_fragment_context.h
index 9ffcb40038c..38db8cbe8ff 100644
--- a/be/src/pipeline/pipeline_fragment_context.h
+++ b/be/src/pipeline/pipeline_fragment_context.h
@@ -78,7 +78,7 @@ public:
RuntimeState* get_runtime_state() { return _runtime_state.get(); }
virtual RuntimeFilterMgr* get_runtime_filter_mgr(UniqueId /*fragment_instance_id*/) {
- return _runtime_state->runtime_filter_mgr();
+ return _runtime_state->local_runtime_filter_mgr();
}
QueryContext* get_query_ctx() { return _query_ctx.get(); }
diff --git a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
index 9087f94db5e..693cb4d5c30 100644
--- a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -210,8 +210,9 @@ Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& r
_runtime_state->set_total_load_streams(request.total_load_streams);
_runtime_state->set_num_local_sink(request.num_local_sink);
- _use_global_rf = request.__isset.parallel_instances && (request.__isset.per_node_shared_scans &&
- !request.per_node_shared_scans.empty());
+ _need_local_merge =
+ request.__isset.parallel_instances &&
+ (request.__isset.per_node_shared_scans && !request.per_node_shared_scans.empty());
// 2. Build pipelines with operators in this fragment.
auto root_pipeline = add_pipeline();
RETURN_IF_ERROR_OR_CATCH_EXCEPTION(_build_pipelines(
@@ -523,11 +524,12 @@ Status PipelineXFragmentContext::_build_pipeline_tasks(
filterparams->query_ctx = _query_ctx.get();
}
- // build runtime_filter_mgr for each instance
+ // build local_runtime_filter_mgr for each instance
runtime_filter_mgr =
std::make_unique<RuntimeFilterMgr>(request.query_id, filterparams.get());
- if (local_params.__isset.runtime_filter_params) {
- runtime_filter_mgr->set_runtime_filter_params(local_params.runtime_filter_params);
+ if (i == 0 && local_params.__isset.runtime_filter_params) {
+ _query_ctx->runtime_filter_mgr()->set_runtime_filter_params(
+ local_params.runtime_filter_params);
}
filterparams->runtime_filter_mgr = runtime_filter_mgr.get();
@@ -986,7 +988,7 @@ Status PipelineXFragmentContext::_create_operator(ObjectPool* pool, const TPlanN
DataSinkOperatorXPtr sink;
sink.reset(new HashJoinBuildSinkOperatorX(pool, next_sink_operator_id(), tnode, descs,
- _use_global_rf));
+ _need_local_merge));
sink->set_dests_id({op->operator_id()});
RETURN_IF_ERROR(build_side_pipe->set_sink(sink));
RETURN_IF_ERROR(build_side_pipe->sink_x()->init(tnode, _runtime_state.get()));
diff --git a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.h b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.h
index 439b0072d72..54714dd4665 100644
--- a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.h
+++ b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.h
@@ -177,7 +177,7 @@ private:
// this is a [n * m] matrix. n is parallelism of pipeline engine and m is the number of pipelines.
std::vector<std::vector<std::unique_ptr<PipelineXTask>>> _tasks;
- bool _use_global_rf = false;
+ bool _need_local_merge = false;
// It is used to manage the lifecycle of RuntimeFilterMergeController
std::vector<std::shared_ptr<RuntimeFilterMergeControllerEntity>> _merge_controller_handlers;
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index a8d29b9f9de..20a78209a36 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -1304,7 +1304,7 @@ Status FragmentMgr::apply_filter(const PPublishFilterRequest* request,
pip_context = iter->second;
DCHECK(pip_context != nullptr);
- runtime_filter_mgr = pip_context->get_runtime_filter_mgr(fragment_instance_id);
+ runtime_filter_mgr = pip_context->get_query_ctx()->runtime_filter_mgr();
} else {
std::unique_lock<std::mutex> lock(_lock);
auto iter = _fragment_instance_map.find(tfragment_instance_id);
@@ -1315,7 +1315,8 @@ Status FragmentMgr::apply_filter(const PPublishFilterRequest* request,
fragment_executor = iter->second;
DCHECK(fragment_executor != nullptr);
- runtime_filter_mgr = fragment_executor->runtime_state()->runtime_filter_mgr();
+ runtime_filter_mgr =
+ fragment_executor->runtime_state()->get_query_ctx()->runtime_filter_mgr();
}
return runtime_filter_mgr->update_filter(request, attach_data);
diff --git a/be/src/runtime/query_context.h b/be/src/runtime/query_context.h
index cf4321beab3..3db91ba2824 100644
--- a/be/src/runtime/query_context.h
+++ b/be/src/runtime/query_context.h
@@ -187,6 +187,9 @@ public:
return _query_options.__isset.fe_process_uuid ? _query_options.fe_process_uuid : 0;
}
+ // global runtime filter mgr, the runtime filter have remote target or
+ // need local merge should regist here. before publish() or push_to_remote()
+ // the runtime filter should do the local merge work
RuntimeFilterMgr* runtime_filter_mgr() { return _runtime_filter_mgr.get(); }
TUniqueId query_id() const { return _query_id; }
diff --git a/be/src/runtime/runtime_filter_mgr.cpp b/be/src/runtime/runtime_filter_mgr.cpp
index 799348ef1d3..95f65c5fc32 100644
--- a/be/src/runtime/runtime_filter_mgr.cpp
+++ b/be/src/runtime/runtime_filter_mgr.cpp
@@ -74,7 +74,7 @@ Status RuntimeFilterMgr::get_consume_filters(const int filter_id,
Status RuntimeFilterMgr::register_consumer_filter(const TRuntimeFilterDesc& desc,
const TQueryOptions& options, int node_id,
IRuntimeFilter** consumer_filter,
- bool build_bf_exactly, bool is_global) {
+ bool build_bf_exactly, bool need_local_merge) {
SCOPED_CONSUME_MEM_TRACKER(_tracker.get());
int32_t key = desc.filter_id;
bool has_exist = false;
@@ -89,28 +89,72 @@ Status RuntimeFilterMgr::register_consumer_filter(const TRuntimeFilterDesc& desc
}
}
- // TODO: union the remote opt and global two case as one case to one judge
- bool remote_opt_or_global = (desc.__isset.opt_remote_rf && desc.opt_remote_rf) || is_global;
-
if (!has_exist) {
IRuntimeFilter* filter;
- RETURN_IF_ERROR(IRuntimeFilter::create(
- _state, remote_opt_or_global ? _state->obj_pool() : &_pool, &desc, &options,
- RuntimeFilterRole::CONSUMER, node_id, &filter, build_bf_exactly, is_global));
+ RETURN_IF_ERROR(IRuntimeFilter::create(_state, &_pool, &desc, &options,
+ RuntimeFilterRole::CONSUMER, node_id, &filter,
+ build_bf_exactly, need_local_merge));
_consumer_map[key].emplace_back(node_id, filter);
*consumer_filter = filter;
- } else if (!remote_opt_or_global) {
+ } else if (!need_local_merge) {
return Status::InvalidArgument("filter has registered");
}
return Status::OK();
}
+Status RuntimeFilterMgr::register_local_merge_producer_filter(
+ const doris::TRuntimeFilterDesc& desc, const doris::TQueryOptions& options,
+ doris::IRuntimeFilter** producer_filter, bool build_bf_exactly) {
+ SCOPED_CONSUME_MEM_TRACKER(_tracker.get());
+ int32_t key = desc.filter_id;
+
+ decltype(_local_merge_producer_map.end()) iter;
+ {
+ std::lock_guard<std::mutex> l(_lock);
+ iter = _local_merge_producer_map.find(key);
+ if (iter == _local_merge_producer_map.end()) {
+ auto [new_iter, _] = _local_merge_producer_map.emplace(key, LocalMergeFilters {});
+ iter = new_iter;
+ }
+ }
+
+ DCHECK(_state != nullptr);
+ RETURN_IF_ERROR(IRuntimeFilter::create(_state, &_pool, &desc, &options,
+ RuntimeFilterRole::PRODUCER, -1, producer_filter,
+ build_bf_exactly, true));
+ {
+ std::lock_guard<std::mutex> l(*iter->second.lock);
+ if (iter->second.filters.empty()) {
+ IRuntimeFilter* merge_filter = nullptr;
+ RETURN_IF_ERROR(IRuntimeFilter::create(_state, &_pool, &desc, &options,
+ RuntimeFilterRole::PRODUCER, -1, &merge_filter,
+ build_bf_exactly, true));
+ iter->second.filters.emplace_back(merge_filter);
+ }
+ iter->second.merge_time++;
+ iter->second.filters.emplace_back(*producer_filter);
+ }
+ return Status::OK();
+}
+
+Status RuntimeFilterMgr::get_local_merge_producer_filters(
+ int filter_id, doris::LocalMergeFilters** local_merge_filters) {
+ std::lock_guard<std::mutex> l(_lock);
+ auto iter = _local_merge_producer_map.find(filter_id);
+ if (iter == _local_merge_producer_map.end()) {
+ return Status::InvalidArgument("unknown filter: {}, role: CONSUMER.", filter_id);
+ }
+ *local_merge_filters = &iter->second;
+ DCHECK(!iter->second.filters.empty());
+ DCHECK_GT(iter->second.merge_time, 0);
+ return Status::OK();
+}
+
Status RuntimeFilterMgr::register_producer_filter(const TRuntimeFilterDesc& desc,
const TQueryOptions& options,
IRuntimeFilter** producer_filter,
- bool build_bf_exactly, bool is_global,
- int parallel_tasks) {
+ bool build_bf_exactly) {
SCOPED_CONSUME_MEM_TRACKER(_tracker.get());
int32_t key = desc.filter_id;
std::lock_guard<std::mutex> l(_lock);
@@ -122,7 +166,7 @@ Status RuntimeFilterMgr::register_producer_filter(const TRuntimeFilterDesc& desc
}
RETURN_IF_ERROR(IRuntimeFilter::create(_state, &_pool, &desc, &options,
RuntimeFilterRole::PRODUCER, -1, producer_filter,
- build_bf_exactly, is_global, parallel_tasks));
+ build_bf_exactly));
_producer_map.emplace(key, *producer_filter);
return Status::OK();
}
@@ -133,7 +177,19 @@ Status RuntimeFilterMgr::update_filter(const PPublishFilterRequest* request,
UpdateRuntimeFilterParams params(request, data, &_pool);
int filter_id = request->filter_id();
std::vector<IRuntimeFilter*> filters;
- RETURN_IF_ERROR(get_consume_filters(filter_id, filters));
+ // The code is organized for upgrade compatibility to prevent infinite waiting
+ // old way update filter the code should be deleted after the upgrade is complete.
+ {
+ std::lock_guard<std::mutex> l(_lock);
+ auto iter = _consumer_map.find(filter_id);
+ if (iter == _consumer_map.end()) {
+ return Status::InvalidArgument("unknown filter: {}, role: CONSUMER.", filter_id);
+ }
+ for (auto& holder : iter->second) {
+ filters.emplace_back(holder.filter);
+ }
+ iter->second.clear();
+ }
for (auto filter : filters) {
RETURN_IF_ERROR(filter->update_filter(¶ms));
}
@@ -143,8 +199,11 @@ Status RuntimeFilterMgr::update_filter(const PPublishFilterRequest* request,
void RuntimeFilterMgr::set_runtime_filter_params(
const TRuntimeFilterParams& runtime_filter_params) {
- this->_merge_addr = runtime_filter_params.runtime_filter_merge_addr;
- this->_has_merge_addr = true;
+ std::lock_guard l(_lock);
+ if (!_has_merge_addr) {
+ _merge_addr = runtime_filter_params.runtime_filter_merge_addr;
+ _has_merge_addr = true;
+ }
}
Status RuntimeFilterMgr::get_merge_addr(TNetworkAddress* addr) {
@@ -454,7 +513,7 @@ RuntimeFilterParamsContext* RuntimeFilterParamsContext::create(RuntimeState* sta
params->runtime_filter_wait_time_ms = state->runtime_filter_wait_time_ms();
params->enable_pipeline_exec = state->enable_pipeline_exec();
params->execution_timeout = state->execution_timeout();
- params->runtime_filter_mgr = state->runtime_filter_mgr();
+ params->runtime_filter_mgr = state->local_runtime_filter_mgr();
params->exec_env = state->exec_env();
params->query_id.set_hi(state->query_id().hi);
params->query_id.set_lo(state->query_id().lo);
@@ -479,8 +538,6 @@ RuntimeFilterParamsContext* RuntimeFilterParamsContext::create(QueryContext* que
params->be_exec_version = query_ctx->be_exec_version();
params->query_ctx = query_ctx;
- params->_obj_pool = &query_ctx->obj_pool;
- params->_is_global = true;
return params;
}
diff --git a/be/src/runtime/runtime_filter_mgr.h b/be/src/runtime/runtime_filter_mgr.h
index 7caea8011d2..c9b455bc107 100644
--- a/be/src/runtime/runtime_filter_mgr.h
+++ b/be/src/runtime/runtime_filter_mgr.h
@@ -53,6 +53,12 @@ class QueryContext;
struct RuntimeFilterParamsContext;
class ExecEnv;
+struct LocalMergeFilters {
+ std::unique_ptr<std::mutex> lock = std::make_unique<std::mutex>();
+ int merge_time = 0;
+ std::vector<IRuntimeFilter*> filters;
+};
+
/// producer:
/// Filter filter;
/// get_filter(filter_id, &filter);
@@ -76,10 +82,18 @@ public:
// register filter
Status register_consumer_filter(const TRuntimeFilterDesc& desc, const TQueryOptions& options,
int node_id, IRuntimeFilter** consumer_filter,
- bool build_bf_exactly = false, bool is_global = false);
+ bool build_bf_exactly = false, bool need_local_merge = false);
+
+ Status register_local_merge_producer_filter(const TRuntimeFilterDesc& desc,
+ const TQueryOptions& options,
+ IRuntimeFilter** producer_filter,
+ bool build_bf_exactly = false);
+
+ Status get_local_merge_producer_filters(int filter_id, LocalMergeFilters** local_merge_filters);
+
Status register_producer_filter(const TRuntimeFilterDesc& desc, const TQueryOptions& options,
- IRuntimeFilter** producer_filter, bool build_bf_exactly = false,
- bool is_global = false, int parallel_tasks = 0);
+ IRuntimeFilter** producer_filter,
+ bool build_bf_exactly = false);
// update filter by remote
Status update_filter(const PPublishFilterRequest* request,
@@ -100,6 +114,7 @@ private:
/// TODO: should it need protected by a mutex?
std::map<int32_t, std::vector<ConsumerFilterHolder>> _consumer_map;
std::map<int32_t, IRuntimeFilter*> _producer_map;
+ std::map<int32_t, LocalMergeFilters> _local_merge_producer_map;
RuntimeFilterParamsContext* _state = nullptr;
std::unique_ptr<MemTracker> _tracker;
@@ -257,15 +272,6 @@ struct RuntimeFilterParamsContext {
int be_exec_version;
QueryContext* query_ctx;
QueryContext* get_query_ctx() const { return query_ctx; }
- ObjectPool* _obj_pool;
- bool _is_global = false;
- PUniqueId fragment_instance_id() const {
- DCHECK(!_is_global);
- return _fragment_instance_id;
- }
- ObjectPool* obj_pool() const {
- DCHECK(_is_global);
- return _obj_pool;
- }
+ PUniqueId fragment_instance_id() const { return _fragment_instance_id; }
};
} // namespace doris
diff --git a/be/src/runtime/runtime_state.cpp b/be/src/runtime/runtime_state.cpp
index d25d914147b..8762366fe04 100644
--- a/be/src/runtime/runtime_state.cpp
+++ b/be/src/runtime/runtime_state.cpp
@@ -101,7 +101,8 @@ RuntimeState::RuntimeState(const TPlanFragmentExecParams& fragment_exec_params,
_runtime_filter_mgr.reset(new RuntimeFilterMgr(fragment_exec_params.query_id,
RuntimeFilterParamsContext::create(this)));
if (fragment_exec_params.__isset.runtime_filter_params) {
- _runtime_filter_mgr->set_runtime_filter_params(fragment_exec_params.runtime_filter_params);
+ _query_ctx->runtime_filter_mgr()->set_runtime_filter_params(
+ fragment_exec_params.runtime_filter_params);
}
}
@@ -305,11 +306,6 @@ Status RuntimeState::init(const TUniqueId& fragment_instance_id, const TQueryOpt
return Status::OK();
}
-void RuntimeState::set_runtime_filter_params(
- const TRuntimeFilterParams& runtime_filter_params) const {
- _runtime_filter_mgr->set_runtime_filter_params(runtime_filter_params);
-}
-
void RuntimeState::init_mem_trackers(const TUniqueId& id, const std::string& name) {
_query_mem_tracker = std::make_shared<MemTrackerLimiter>(
MemTrackerLimiter::Type::EXPERIMENTAL, fmt::format("{}#Id={}", name, print_id(id)));
@@ -506,4 +502,32 @@ bool RuntimeState::enable_page_cache() const {
(_query_options.__isset.enable_page_cache && _query_options.enable_page_cache);
}
+RuntimeFilterMgr* RuntimeState::global_runtime_filter_mgr() {
+ return _query_ctx->runtime_filter_mgr();
+}
+
+Status RuntimeState::register_producer_runtime_filter(const doris::TRuntimeFilterDesc& desc,
+ bool need_local_merge,
+ doris::IRuntimeFilter** producer_filter,
+ bool build_bf_exactly) {
+ if (desc.has_remote_targets || need_local_merge) {
+ return global_runtime_filter_mgr()->register_local_merge_producer_filter(
+ desc, query_options(), producer_filter, build_bf_exactly);
+ } else {
+ return local_runtime_filter_mgr()->register_producer_filter(
+ desc, query_options(), producer_filter, build_bf_exactly);
+ }
+}
+
+Status RuntimeState::register_consumer_runtime_filter(const doris::TRuntimeFilterDesc& desc,
+ bool need_local_merge, int node_id,
+ doris::IRuntimeFilter** consumer_filter) {
+ if (desc.has_remote_targets || need_local_merge) {
+ return global_runtime_filter_mgr()->register_consumer_filter(desc, query_options(), node_id,
+ consumer_filter, false, true);
+ } else {
+ return local_runtime_filter_mgr()->register_consumer_filter(desc, query_options(), node_id,
+ consumer_filter, false, false);
+ }
+}
} // end namespace doris
diff --git a/be/src/runtime/runtime_state.h b/be/src/runtime/runtime_state.h
index cdc7e83f042..03b518e5c34 100644
--- a/be/src/runtime/runtime_state.h
+++ b/be/src/runtime/runtime_state.h
@@ -44,6 +44,7 @@
#include "util/runtime_profile.h"
namespace doris {
+class IRuntimeFilter;
namespace pipeline {
class PipelineXLocalStateBase;
@@ -98,8 +99,6 @@ public:
Status init(const TUniqueId& fragment_instance_id, const TQueryOptions& query_options,
const TQueryGlobals& query_globals, ExecEnv* exec_env);
- void set_runtime_filter_params(const TRuntimeFilterParams& runtime_filter_params) const;
-
// for ut and non-query.
void set_exec_env(ExecEnv* exec_env) { _exec_env = exec_env; }
void init_mem_trackers(const TUniqueId& id = TUniqueId(), const std::string& name = "unknown");
@@ -454,7 +453,10 @@ public:
// if load mem limit is not set, or is zero, using query mem limit instead.
int64_t get_load_mem_limit();
- RuntimeFilterMgr* runtime_filter_mgr() {
+ // local runtime filter mgr, the runtime filter do not have remote target or
+ // not need local merge should regist here. the instance exec finish, the local
+ // runtime filter mgr can release the memory of local runtime filter
+ RuntimeFilterMgr* local_runtime_filter_mgr() {
if (_pipeline_x_runtime_filter_mgr) {
return _pipeline_x_runtime_filter_mgr;
} else {
@@ -462,6 +464,8 @@ public:
}
}
+ RuntimeFilterMgr* global_runtime_filter_mgr();
+
void set_pipeline_x_runtime_filter_mgr(RuntimeFilterMgr* pipeline_x_runtime_filter_mgr) {
_pipeline_x_runtime_filter_mgr = pipeline_x_runtime_filter_mgr;
}
@@ -567,6 +571,15 @@ public:
return _task_execution_context;
}
+ Status register_producer_runtime_filter(const doris::TRuntimeFilterDesc& desc,
+ bool need_local_merge,
+ doris::IRuntimeFilter** producer_filter,
+ bool build_bf_exactly);
+
+ Status register_consumer_runtime_filter(const doris::TRuntimeFilterDesc& desc,
+ bool need_local_merge, int node_id,
+ doris::IRuntimeFilter** producer_filter);
+
private:
Status create_error_log_file();
@@ -595,9 +608,6 @@ private:
// owned by PipelineXFragmentContext
RuntimeFilterMgr* _pipeline_x_runtime_filter_mgr = nullptr;
- // Protects _data_stream_recvrs_pool
- std::mutex _data_stream_recvrs_lock;
-
// Data stream receivers created by a plan fragment are gathered here to make sure
// they are destroyed before _obj_pool (class members are destroyed in reverse order).
// Receivers depend on the descriptor table and we need to guarantee that their control
diff --git a/be/src/vec/exec/join/vhash_join_node.cpp b/be/src/vec/exec/join/vhash_join_node.cpp
index 67dffa4b203..de3b63371ee 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -180,9 +180,9 @@ Status HashJoinNode::init(const TPlanNode& tnode, RuntimeState* state) {
#endif
for (size_t i = 0; i < _runtime_filter_descs.size(); i++) {
- RETURN_IF_ERROR(state->runtime_filter_mgr()->register_producer_filter(
- _runtime_filter_descs[i], state->query_options(), &_runtime_filters[i],
- _probe_expr_ctxs.size() == 1));
+ RETURN_IF_ERROR(state->register_producer_runtime_filter(_runtime_filter_descs[i], false,
+ &_runtime_filters[i],
+ _probe_expr_ctxs.size() == 1));
}
// init left/right output slots flags, only column of slot_id in _hash_output_slot_ids need
diff --git a/be/src/vec/exec/join/vnested_loop_join_node.cpp b/be/src/vec/exec/join/vnested_loop_join_node.cpp
index 150068096ba..ad168ba9c86 100644
--- a/be/src/vec/exec/join/vnested_loop_join_node.cpp
+++ b/be/src/vec/exec/join/vnested_loop_join_node.cpp
@@ -118,8 +118,8 @@ Status VNestedLoopJoinNode::init(const TPlanNode& tnode, RuntimeState* state) {
std::vector<TExpr> filter_src_exprs;
for (size_t i = 0; i < _runtime_filter_descs.size(); i++) {
filter_src_exprs.push_back(_runtime_filter_descs[i].src_expr);
- RETURN_IF_ERROR(state->runtime_filter_mgr()->register_producer_filter(
- _runtime_filter_descs[i], state->query_options(), &_runtime_filters[i]));
+ RETURN_IF_ERROR(state->register_producer_runtime_filter(_runtime_filter_descs[i], false,
+ &_runtime_filters[i], false));
}
RETURN_IF_ERROR(vectorized::VExpr::create_expr_trees(filter_src_exprs, _filter_src_expr_ctxs));
return Status::OK();
diff --git a/be/src/vec/exec/runtime_filter_consumer.cpp b/be/src/vec/exec/runtime_filter_consumer.cpp
index eba11dac45d..e683c4f2be0 100644
--- a/be/src/vec/exec/runtime_filter_consumer.cpp
+++ b/be/src/vec/exec/runtime_filter_consumer.cpp
@@ -30,9 +30,9 @@ RuntimeFilterConsumer::RuntimeFilterConsumer(const int32_t filter_id,
_blocked_by_rf = std::make_shared<std::atomic_bool>(false);
}
-Status RuntimeFilterConsumer::init(RuntimeState* state, bool is_global) {
+Status RuntimeFilterConsumer::init(RuntimeState* state, bool need_local_merge) {
_state = state;
- RETURN_IF_ERROR(_register_runtime_filter(is_global));
+ RETURN_IF_ERROR(_register_runtime_filter(need_local_merge));
return Status::OK();
}
@@ -45,28 +45,15 @@ void RuntimeFilterConsumer::_init_profile(RuntimeProfile* profile) {
profile->add_info_string("RuntimeFilters: ", ss.str());
}
-Status RuntimeFilterConsumer::_register_runtime_filter(bool is_global) {
+Status RuntimeFilterConsumer::_register_runtime_filter(bool need_local_merge) {
int filter_size = _runtime_filter_descs.size();
_runtime_filter_ctxs.reserve(filter_size);
_runtime_filter_ready_flag.reserve(filter_size);
for (int i = 0; i < filter_size; ++i) {
IRuntimeFilter* runtime_filter = nullptr;
const auto& filter_desc = _runtime_filter_descs[i];
- if (filter_desc.__isset.opt_remote_rf && filter_desc.opt_remote_rf) {
- DCHECK(filter_desc.has_remote_targets);
- RETURN_IF_ERROR(_state->get_query_ctx()->runtime_filter_mgr()->register_consumer_filter(
- filter_desc, _state->query_options(), _filter_id, &runtime_filter, false,
- is_global));
- } else if (is_global) {
- // For pipelineX engine, runtime filter is global iff data distribution is ignored.
- RETURN_IF_ERROR(_state->get_query_ctx()->runtime_filter_mgr()->register_consumer_filter(
- filter_desc, _state->query_options(), _filter_id, &runtime_filter, false,
- is_global));
- } else {
- RETURN_IF_ERROR(_state->runtime_filter_mgr()->register_consumer_filter(
- filter_desc, _state->query_options(), _filter_id, &runtime_filter, false,
- is_global));
- }
+ RETURN_IF_ERROR(_state->register_consumer_runtime_filter(filter_desc, need_local_merge,
+ _filter_id, &runtime_filter));
_runtime_filter_ctxs.emplace_back(runtime_filter);
_runtime_filter_ready_flag.emplace_back(false);
}
diff --git a/be/src/vec/exec/runtime_filter_consumer.h b/be/src/vec/exec/runtime_filter_consumer.h
index 15b9455ac56..b8513e666bc 100644
--- a/be/src/vec/exec/runtime_filter_consumer.h
+++ b/be/src/vec/exec/runtime_filter_consumer.h
@@ -30,7 +30,7 @@ public:
const RowDescriptor& row_descriptor, VExprContextSPtrs& conjuncts);
~RuntimeFilterConsumer() = default;
- Status init(RuntimeState* state, bool is_global = false);
+ Status init(RuntimeState* state, bool need_local_merge = false);
// Try to append late arrived runtime filters.
// Return num of filters which are applied already.
@@ -42,7 +42,7 @@ public:
protected:
// Register and get all runtime filters at Init phase.
- Status _register_runtime_filter(bool is_global);
+ Status _register_runtime_filter(bool need_local_merge);
// Get all arrived runtime filters at Open phase.
Status _acquire_runtime_filter();
// Append late-arrival runtime filters to the vconjunct_ctx.
diff --git a/be/src/vec/exec/vdata_gen_scan_node.cpp b/be/src/vec/exec/vdata_gen_scan_node.cpp
index 42f6250a030..13d19921b03 100644
--- a/be/src/vec/exec/vdata_gen_scan_node.cpp
+++ b/be/src/vec/exec/vdata_gen_scan_node.cpp
@@ -81,13 +81,8 @@ Status VDataGenFunctionScanNode::prepare(RuntimeState* state) {
// TODO: use runtime filter to filte result block, maybe this node need derive from vscan_node.
for (const auto& filter_desc : _runtime_filter_descs) {
IRuntimeFilter* runtime_filter = nullptr;
- if (filter_desc.__isset.opt_remote_rf && filter_desc.opt_remote_rf) {
- RETURN_IF_ERROR(state->get_query_ctx()->runtime_filter_mgr()->register_consumer_filter(
- filter_desc, state->query_options(), id(), &runtime_filter, false));
- } else {
- RETURN_IF_ERROR(state->runtime_filter_mgr()->register_consumer_filter(
- filter_desc, state->query_options(), id(), &runtime_filter, false));
- }
+ RETURN_IF_ERROR(
+ state->register_consumer_runtime_filter(filter_desc, false, id(), &runtime_filter));
runtime_filter->init_profile(_runtime_profile.get());
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
index 333b73c217c..d0322e375a9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -2150,7 +2150,8 @@ public class Coordinator implements CoordInterface {
}
for (RuntimeFilterId rid : fragment.getBuilderRuntimeFilterIds()) {
- ridToBuilderNum.merge(rid, params.instanceExecParams.size(), Integer::sum);
+ ridToBuilderNum.merge(rid,
+ (int) params.instanceExecParams.stream().map(ins -> ins.host).distinct().count(), Integer::sum);
}
}
// Use the uppermost fragment as a merged node, the uppermost fragment has one and only one instance
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 15/19: [refine](pipelinex) get sink local state does not require an id. #31195
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 90a7f04349afdada593f6209b5bdf56c41e43d8a
Author: Mryange <59...@users.noreply.github.com>
AuthorDate: Thu Feb 22 10:01:56 2024 +0800
[refine](pipelinex) get sink local state does not require an id. #31195
---
be/src/pipeline/exec/hashjoin_build_sink.h | 2 +-
be/src/pipeline/pipeline_x/operator.cpp | 5 ++---
be/src/pipeline/pipeline_x/operator.h | 10 ++++++----
be/src/pipeline/pipeline_x/pipeline_x_task.cpp | 8 ++++----
be/src/runtime/runtime_state.cpp | 6 +++---
be/src/runtime/runtime_state.h | 4 ++--
6 files changed, 18 insertions(+), 17 deletions(-)
diff --git a/be/src/pipeline/exec/hashjoin_build_sink.h b/be/src/pipeline/exec/hashjoin_build_sink.h
index eb78d6bf515..080097d8bc4 100644
--- a/be/src/pipeline/exec/hashjoin_build_sink.h
+++ b/be/src/pipeline/exec/hashjoin_build_sink.h
@@ -139,7 +139,7 @@ public:
SourceState source_state) override;
bool should_dry_run(RuntimeState* state) override {
- return _is_broadcast_join && !state->get_sink_local_state(operator_id())
+ return _is_broadcast_join && !state->get_sink_local_state()
->cast<HashJoinBuildSinkLocalState>()
._should_build_hash_table;
}
diff --git a/be/src/pipeline/pipeline_x/operator.cpp b/be/src/pipeline/pipeline_x/operator.cpp
index 101b02cf406..19cda4d51ad 100644
--- a/be/src/pipeline/pipeline_x/operator.cpp
+++ b/be/src/pipeline/pipeline_x/operator.cpp
@@ -235,7 +235,7 @@ std::string DataSinkOperatorXBase::debug_string(int indentation_level) const {
}
std::string DataSinkOperatorXBase::debug_string(RuntimeState* state, int indentation_level) const {
- return state->get_sink_local_state(operator_id())->debug_string(indentation_level);
+ return state->get_sink_local_state()->debug_string(indentation_level);
}
Status DataSinkOperatorXBase::init(const TDataSink& tsink) {
@@ -498,8 +498,7 @@ Status StreamingOperatorX<LocalStateType>::get_block(RuntimeState* state, vector
template <typename LocalStateType>
Status StatefulOperatorX<LocalStateType>::get_block(RuntimeState* state, vectorized::Block* block,
SourceState& source_state) {
- auto& local_state = state->get_local_state(OperatorX<LocalStateType>::operator_id())
- ->template cast<LocalStateType>();
+ auto& local_state = get_local_state(state);
if (need_more_input_data(state)) {
local_state._child_block->clear_column_data();
RETURN_IF_ERROR(OperatorX<LocalStateType>::_child_x->get_block_after_projects(
diff --git a/be/src/pipeline/pipeline_x/operator.h b/be/src/pipeline/pipeline_x/operator.h
index 87101de0612..387fddef059 100644
--- a/be/src/pipeline/pipeline_x/operator.h
+++ b/be/src/pipeline/pipeline_x/operator.h
@@ -536,8 +536,8 @@ public:
[[nodiscard]] bool is_source() const override { return false; }
- Status close(RuntimeState* state, Status exec_status) {
- auto result = state->get_sink_local_state_result(operator_id());
+ static Status close(RuntimeState* state, Status exec_status) {
+ auto result = state->get_sink_local_state_result();
if (!result) {
return result.error();
}
@@ -600,7 +600,7 @@ public:
using LocalState = LocalStateType;
[[nodiscard]] LocalState& get_local_state(RuntimeState* state) const {
- return state->get_sink_local_state(operator_id())->template cast<LocalState>();
+ return state->get_sink_local_state()->template cast<LocalState>();
}
};
@@ -663,8 +663,10 @@ public:
: OperatorX<LocalStateType>(pool, tnode, operator_id, descs) {}
virtual ~StatefulOperatorX() = default;
+ using OperatorX<LocalStateType>::get_local_state;
+
[[nodiscard]] Status get_block(RuntimeState* state, vectorized::Block* block,
- SourceState& source_state) override;
+ SourceState& source_state) final;
[[nodiscard]] virtual Status pull(RuntimeState* state, vectorized::Block* block,
SourceState& source_state) const = 0;
diff --git a/be/src/pipeline/pipeline_x/pipeline_x_task.cpp b/be/src/pipeline/pipeline_x/pipeline_x_task.cpp
index 7579789534f..d4117661a79 100644
--- a/be/src/pipeline/pipeline_x/pipeline_x_task.cpp
+++ b/be/src/pipeline/pipeline_x/pipeline_x_task.cpp
@@ -91,9 +91,9 @@ Status PipelineXTask::prepare(const TPipelineInstanceParams& local_params, const
std::vector<TScanRangeParams> no_scan_ranges;
auto scan_ranges = find_with_default(local_params.per_node_scan_ranges,
_operators.front()->node_id(), no_scan_ranges);
- auto* parent_profile = _state->get_sink_local_state(_sink->operator_id())->profile();
+ auto* parent_profile = _state->get_sink_local_state()->profile();
query_ctx->register_query_statistics(
- _state->get_sink_local_state(_sink->operator_id())->get_query_statistics_ptr());
+ _state->get_sink_local_state()->get_query_statistics_ptr());
for (int op_idx = _operators.size() - 1; op_idx >= 0; op_idx--) {
auto& op = _operators[op_idx];
@@ -135,7 +135,7 @@ Status PipelineXTask::_extract_dependencies() {
}
}
{
- auto* local_state = _state->get_sink_local_state(_sink->operator_id());
+ auto* local_state = _state->get_sink_local_state();
auto* dep = local_state->dependency();
DCHECK(dep != nullptr);
_write_dependencies = dep;
@@ -206,7 +206,7 @@ Status PipelineXTask::_open() {
RETURN_IF_ERROR(st);
}
}
- RETURN_IF_ERROR(_state->get_sink_local_state(_sink->operator_id())->open(_state));
+ RETURN_IF_ERROR(_state->get_sink_local_state()->open(_state));
_opened = true;
return Status::OK();
}
diff --git a/be/src/runtime/runtime_state.cpp b/be/src/runtime/runtime_state.cpp
index 8762366fe04..5a32b37c40f 100644
--- a/be/src/runtime/runtime_state.cpp
+++ b/be/src/runtime/runtime_state.cpp
@@ -486,13 +486,13 @@ void RuntimeState::emplace_sink_local_state(
_sink_local_state = std::move(state);
}
-doris::pipeline::PipelineXSinkLocalStateBase* RuntimeState::get_sink_local_state(int) {
+doris::pipeline::PipelineXSinkLocalStateBase* RuntimeState::get_sink_local_state() {
return _sink_local_state.get();
}
-Result<RuntimeState::SinkLocalState*> RuntimeState::get_sink_local_state_result(int id) {
+Result<RuntimeState::SinkLocalState*> RuntimeState::get_sink_local_state_result() {
if (!_sink_local_state) {
- return ResultError(Status::InternalError("_op_id_to_sink_local_state id:{} is null", id));
+ return ResultError(Status::InternalError("_op_id_to_sink_local_state not exist"));
}
return _sink_local_state.get();
}
diff --git a/be/src/runtime/runtime_state.h b/be/src/runtime/runtime_state.h
index 03b518e5c34..bd54ca98e66 100644
--- a/be/src/runtime/runtime_state.h
+++ b/be/src/runtime/runtime_state.h
@@ -552,9 +552,9 @@ public:
void emplace_sink_local_state(int id, std::unique_ptr<SinkLocalState> state);
- SinkLocalState* get_sink_local_state(int id);
+ SinkLocalState* get_sink_local_state();
- Result<SinkLocalState*> get_sink_local_state_result(int id);
+ Result<SinkLocalState*> get_sink_local_state_result();
void resize_op_id_to_local_state(int operator_size);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 09/19: [fix](audit) The time field in the audit log table is set to the millisecond level (#31096)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 70dc40ab3c8d6926029e81963179038d763794dc
Author: lw112 <13...@users.noreply.github.com>
AuthorDate: Thu Feb 22 00:01:42 2024 +0800
[fix](audit) The time field in the audit log table is set to the millisecond level (#31096)
---
fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchema.java | 2 +-
.../src/main/java/org/apache/doris/plugin/audit/AuditLoaderPlugin.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchema.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchema.java
index 7d348761704..5a4934ec2a4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchema.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchema.java
@@ -68,7 +68,7 @@ public class InternalSchema {
// audit table
AUDIT_SCHEMA = new ArrayList<>();
AUDIT_SCHEMA.add(new ColumnDef("query_id", TypeDef.createVarchar(48), true));
- AUDIT_SCHEMA.add(new ColumnDef("time", TypeDef.create(PrimitiveType.DATETIME), true));
+ AUDIT_SCHEMA.add(new ColumnDef("time", TypeDef.createDatetimeV2(3), true));
AUDIT_SCHEMA.add(new ColumnDef("client_ip", TypeDef.createVarchar(128), true));
AUDIT_SCHEMA.add(new ColumnDef("user", TypeDef.createVarchar(128), true));
AUDIT_SCHEMA.add(new ColumnDef("catalog", TypeDef.createVarchar(128), true));
diff --git a/fe/fe-core/src/main/java/org/apache/doris/plugin/audit/AuditLoaderPlugin.java b/fe/fe-core/src/main/java/org/apache/doris/plugin/audit/AuditLoaderPlugin.java
index d8941605c35..f5096d9a8f6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/plugin/audit/AuditLoaderPlugin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/plugin/audit/AuditLoaderPlugin.java
@@ -148,7 +148,7 @@ public class AuditLoaderPlugin extends Plugin implements AuditPlugin {
private void fillLogBuffer(AuditEvent event, StringBuilder logBuffer) {
logBuffer.append(event.queryId).append("\t");
- logBuffer.append(TimeUtils.longToTimeString(event.timestamp)).append("\t");
+ logBuffer.append(TimeUtils.longToTimeStringWithms(event.timestamp)).append("\t");
logBuffer.append(event.clientIp).append("\t");
logBuffer.append(event.user).append("\t");
logBuffer.append(event.ctl).append("\t");
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 18/19: [fix](invert index) fix the error issue in the unit test remove_element_only_in_table (#31238)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 90ab5ec2d993866027236e78187d048cce64df75
Author: zzzxl <33...@users.noreply.github.com>
AuthorDate: Thu Feb 22 11:54:21 2024 +0800
[fix](invert index) fix the error issue in the unit test remove_element_only_in_table (#31238)
---
be/test/olap/rowset/segment_v2/inverted_index_searcher_cache_test.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/test/olap/rowset/segment_v2/inverted_index_searcher_cache_test.cpp b/be/test/olap/rowset/segment_v2/inverted_index_searcher_cache_test.cpp
index b11aa1fc78a..8475d85939c 100644
--- a/be/test/olap/rowset/segment_v2/inverted_index_searcher_cache_test.cpp
+++ b/be/test/olap/rowset/segment_v2/inverted_index_searcher_cache_test.cpp
@@ -324,7 +324,7 @@ TEST_F(InvertedIndexSearcherCacheTest, remove_element_only_in_table) {
EXPECT_TRUE(status);
auto cache_value_use_cache = inverted_index_cache_handle.get_index_cache_value();
- EXPECT_LE(UnixMillis(), cache_value_use_cache->last_visit_time);
+ EXPECT_GE(UnixMillis(), cache_value_use_cache->last_visit_time);
}
delete index_searcher_cache;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org