You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "dataroaring (via GitHub)" <gi...@apache.org> on 2023/11/05 09:08:00 UTC
[PR] Row distribution [doris]
dataroaring opened a new pull request, #26432:
URL: https://github.com/apache/doris/pull/26432
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1797733872
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1797770670
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1793735889
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1799017800
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.42 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162169087 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1797992217
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1797811646
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1797950227
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798382872
TeamCity be ut coverage result:
Function Coverage: 37.03% (8400/22686)
Line Coverage: 29.46% (68002/230803)
Region Coverage: 28.13% (35155/124974)
Branch Coverage: 24.92% (17963/72094)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cba33804750b97199b9283458adb5fc6a0b752c7_cba33804750b97199b9283458adb5fc6a0b752c7/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1794800573
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798253554
TeamCity be ut coverage result:
Function Coverage: 36.99% (8398/22702)
Line Coverage: 29.45% (68015/230982)
Region Coverage: 28.12% (35169/125078)
Branch Coverage: 24.90% (17969/72154)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f2729e1a56584c329ded037d927da2bc0ef094e0_f2729e1a56584c329ded037d927da2bc0ef094e0/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798353830
TeamCity be ut coverage result:
Function Coverage: 37.00% (8399/22702)
Line Coverage: 29.44% (68002/230983)
Region Coverage: 28.11% (35163/125078)
Branch Coverage: 24.90% (17967/72154)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1193066cd9004d87972034e561bae7a8d177d880_1193066cd9004d87972034e561bae7a8d177d880/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1793742965
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.42 seconds
stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 56.0 seconds inserted 10000000 Rows, about 178K ops/s
storage size: 17162203014 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26432:
URL: https://github.com/apache/doris/pull/26432#discussion_r1382573855
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include <gen_cpp/FrontendService.h>
Review Comment:
warning: 'gen_cpp/FrontendService.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/FrontendService.h>
^
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+using Payload = std::pair<std::unique_ptr<vectorized::IColumn::Selector>, std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr; // not owned, set when open
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {
+ }
Review Comment:
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
```suggestion
VRowDistribution() = default;
```
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/sink/vrow_distribution.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
+ vectorized::DataTypePtr value_type,
+ std::vector<int64_t> filter) {
+ _partitions_need_create.clear();
+ std::set<std::string> deduper;
+ // de-duplication
+ for (auto row : filter) {
+ deduper.emplace(value_type->to_string(*col, row));
+ }
+ for (auto& value : deduper) {
+ TStringLiteral node;
+ node.value = value;
+ _partitions_need_create.emplace_back(std::vector {node}); // only 1 partition column now
+ }
+}
+
+Status VRowDistribution::_automatic_create_partition() {
+ SCOPED_TIMER(_add_partition_request_timer);
+ TCreatePartitionRequest request;
+ TCreatePartitionResult result;
+ request.__set_txn_id(_txn_id);
+ request.__set_db_id(_vpartition->db_id());
+ request.__set_table_id(_vpartition->table_id());
+ request.__set_partitionValues(_partitions_need_create);
+
+ VLOG(1) << "automatic partition rpc begin request " << request;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ int time_out = _state->execution_timeout() * 1000;
Review Comment:
warning: 1000 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
```cpp
int time_out = _state->execution_timeout() * 1000;
^
```
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/sink/vrow_distribution.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
Review Comment:
warning: method '_save_missing_values' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+using Payload = std::pair<std::unique_ptr<vectorized::IColumn::Selector>, std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr; // not owned, set when open
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {
+ }
+
+ void init(VRowDistributionContext *ctx) {
+ _state = ctx->state;
+ _channels = ctx->channels;
+ _block_convertor = ctx->block_convertor;
+ _tablet_finder = ctx->tablet_finder;
+ _vpartition = ctx->vpartition;
+ _add_partition_request_timer = ctx->add_partition_request_timer;
+ _txn_id = ctx->txn_id;
+ _pool = ctx->pool;
+ _location = ctx->location;
+ _vec_output_expr_ctxs = ctx->vec_output_expr_ctxs;
+ _on_partitions_created = ctx->on_partitions_created;
+ }
+
+ using ChannelDistributionPayload = std::vector<std::unordered_map<VNodeChannel*, Payload>>;
+
+ Status generate_rows_distribution(vectorized::Block& input_block,
+ std::shared_ptr<vectorized::Block>& block,
+ int64_t& filtered_rows, bool& has_filtered_rows,
+ ChannelDistributionPayload& channel_to_payload);
+
+private:
+ std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr> _get_partition_function();
+ void _save_missing_values(vectorized::ColumnPtr col, vectorized::DataTypePtr value_type,
+ std::vector<int64_t> filter);
+
+ // create partitions when need for auto-partition table using #_partitions_need_create.
+ Status _automatic_create_partition();
+
+ Status _single_partition_generate(vectorized::Block* block,
+ ChannelDistributionPayload& channel_to_payload,
+ size_t num_rows, bool has_filtered_rows);
+
+ void _generate_row_distribution_payload(
+ ChannelDistributionPayload& channel_to_payload,
+ const std::vector<VOlapTablePartition*>& partitions,
+ const std::vector<uint32_t>& tablet_indexes, const std::vector<bool>& skip,
+ size_t row_cnt);
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/vec/sink/vrow_distribution.h:85:** previously declared here
```cpp
private:
^
```
</details>
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+using Payload = std::pair<std::unique_ptr<vectorized::IColumn::Selector>, std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
Review Comment:
warning: use 'using' instead of 'typedef' [modernize-use-using]
```suggestion
using OnPartitionsCreated = Status (*)(void *, int *);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26432:
URL: https://github.com/apache/doris/pull/26432#discussion_r1383347238
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,288 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/sink/vrow_distribution.h"
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
+ vectorized::DataTypePtr value_type) {
+ _partitions_need_create.clear();
+ std::set<std::string> deduper;
+ // de-duplication
+ for (auto row : _missing_map) {
+ deduper.emplace(value_type->to_string(*col, row));
+ }
+ for (auto& value : deduper) {
+ TStringLiteral node;
+ node.value = value;
+ _partitions_need_create.emplace_back(std::vector {node}); // only 1 partition column now
+ }
+}
+
+Status VRowDistribution::_automatic_create_partition() {
+ SCOPED_TIMER(_add_partition_request_timer);
+ TCreatePartitionRequest request;
+ TCreatePartitionResult result;
+ request.__set_txn_id(_txn_id);
+ request.__set_db_id(_vpartition->db_id());
+ request.__set_table_id(_vpartition->table_id());
+ request.__set_partitionValues(_partitions_need_create);
+
+ VLOG(1) << "automatic partition rpc begin request " << request;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ int time_out = _state->execution_timeout() * 1000;
+ RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
+ master_addr.hostname, master_addr.port,
+ [&request, &result](FrontendServiceConnection& client) {
+ client->createPartition(result, request);
+ },
+ time_out));
+
+ Status status(Status::create(result.status));
+ VLOG(1) << "automatic partition rpc end response " << result;
+ if (result.status.status_code == TStatusCode::OK) {
+ // add new created partitions
+ RETURN_IF_ERROR(_vpartition->add_partitions(result.partitions));
+ RETURN_IF_ERROR(_on_partitions_created(_caller, &result));
+ }
+
+ return status;
+}
+
+void VRowDistribution::_get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids) {
+ tablet_ids.reserve(block->rows());
+ for (int row_idx = 0; row_idx < block->rows(); row_idx++) {
+ if (_skip[row_idx]) {
+ continue;
+ }
+ auto& partition = _partitions[row_idx];
+ auto& tablet_index = _tablet_indexes[row_idx];
+ auto& index = partition->indexes[index_idx];
+
+ auto tablet_id = index.tablets[tablet_index];
+ tablet_ids[row_idx] = tablet_id;
+ }
+}
+
+void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
+ RowPartTabletTuple& row_part_tablet_tuple) {
+ auto& row_ids = std::get<0>(row_part_tablet_tuple);
+ auto& partition_ids = std::get<1>(row_part_tablet_tuple);
+ auto& tablet_ids = std::get<2>(row_part_tablet_tuple);
+
+ for (size_t i = 0; i < block->rows(); i++) {
+ if (!_skip[i]) {
+ row_ids.emplace_back(i);
+ partition_ids.emplace_back(_partitions[i]->id);
+ tablet_ids.emplace_back(_tablet_ids[i]);
+ }
+ }
+}
+
+Status VRowDistribution::_filter_block_by_skip_and_where_clause(
Review Comment:
warning: method '_filter_block_by_skip_and_where_clause' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status VRowDistribution::_filter_block_by_skip_and_where_clause(
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+using RowPartTabletTuple =
+ std::tuple<std::vector<int64_t>, std::vector<int64_t>, std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
Review Comment:
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
```suggestion
VRowDistribution() = default;
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+using RowPartTabletTuple =
+ std::tuple<std::vector<int64_t>, std::vector<int64_t>, std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
+ virtual ~VRowDistribution() {}
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```suggestion
virtual ~VRowDistribution() = default;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1800885573
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1793732470
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1793679737
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1793687119
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.89 seconds
stream load tsv: 590 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 58.0 seconds inserted 10000000 Rows, about 172K ops/s
storage size: 17162338866 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26432:
URL: https://github.com/apache/doris/pull/26432#discussion_r1382576658
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+using Payload = std::pair<std::unique_ptr<vectorized::IColumn::Selector>, std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr; // not owned, set when open
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {
+ }
+
+ void init(VRowDistributionContext *ctx) {
+ _state = ctx->state;
+ _channels = ctx->channels;
+ _block_convertor = ctx->block_convertor;
+ _tablet_finder = ctx->tablet_finder;
+ _vpartition = ctx->vpartition;
+ _add_partition_request_timer = ctx->add_partition_request_timer;
+ _txn_id = ctx->txn_id;
+ _pool = ctx->pool;
+ _location = ctx->location;
+ _vec_output_expr_ctxs = ctx->vec_output_expr_ctxs;
+ _on_partitions_created = ctx->on_partitions_created;
+ _caller = ctx->caller;
+ }
+
+ using ChannelDistributionPayload = std::vector<std::unordered_map<VNodeChannel*, Payload>>;
+
+ Status generate_rows_distribution(vectorized::Block& input_block,
+ std::shared_ptr<vectorized::Block>& block,
+ int64_t& filtered_rows, bool& has_filtered_rows,
+ ChannelDistributionPayload& channel_to_payload);
+
+private:
+ std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr> _get_partition_function();
+ void _save_missing_values(vectorized::ColumnPtr col, vectorized::DataTypePtr value_type,
+ std::vector<int64_t> filter);
+
+ // create partitions when need for auto-partition table using #_partitions_need_create.
+ Status _automatic_create_partition();
+
+ Status _single_partition_generate(vectorized::Block* block,
+ ChannelDistributionPayload& channel_to_payload,
+ size_t num_rows, bool has_filtered_rows);
+
+ void _generate_row_distribution_payload(
+ ChannelDistributionPayload& channel_to_payload,
+ const std::vector<VOlapTablePartition*>& partitions,
+ const std::vector<uint32_t>& tablet_indexes, const std::vector<bool>& skip,
+ size_t row_cnt);
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/vec/sink/vrow_distribution.h:87:** previously declared here
```cpp
private:
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798222808
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26432:
URL: https://github.com/apache/doris/pull/26432#discussion_r1384374326
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+using RowPartTabletTuple =
+ std::tuple<std::vector<int64_t>, std::vector<int64_t>, std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
+ virtual ~VRowDistribution() {}
+
Review Comment:
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
```suggestion
virtual ~VRowDistribution() = default;
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+using RowPartTabletTuple =
+ std::tuple<std::vector<int64_t>, std::vector<int64_t>, std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
+ virtual ~VRowDistribution() {}
Review Comment:
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
```suggestion
VRowDistribution() = default;
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+using RowPartTabletTuple =
+ std::tuple<std::vector<int64_t>, std::vector<int64_t>, std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
Review Comment:
warning: use 'using' instead of 'typedef' [modernize-use-using]
```suggestion
using OnPartitionsCreated = Status (*)(void *, TCreatePartitionResult *);
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+using RowPartTabletTuple =
+ std::tuple<std::vector<int64_t>, std::vector<int64_t>, std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
+ virtual ~VRowDistribution() {}
+
+ void init(VRowDistributionContext* ctx) {
+ _state = ctx->state;
+ _channels = ctx->channels;
+ _block_convertor = ctx->block_convertor;
+ _tablet_finder = ctx->tablet_finder;
+ _vpartition = ctx->vpartition;
+ _add_partition_request_timer = ctx->add_partition_request_timer;
+ _txn_id = ctx->txn_id;
+ _pool = ctx->pool;
+ _location = ctx->location;
+ _vec_output_expr_ctxs = ctx->vec_output_expr_ctxs;
+ _on_partitions_created = ctx->on_partitions_created;
+ _caller = ctx->caller;
+ _schema = ctx->schema;
+ }
+
+ // auto partition
+ // mv where clause
+ // v1 needs index->node->row_ids - tabletids
+ // v2 needs index,tablet->rowids
+ Status generate_rows_distribution(vectorized::Block& input_block,
+ std::shared_ptr<vectorized::Block>& block,
+ int64_t& filtered_rows, bool& has_filtered_rows,
+ std::vector<RowPartTabletTuple>& row_part_tablet_tuples);
+
+private:
+ std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr> _get_partition_function();
+
+ void _save_missing_values(vectorized::ColumnPtr col, vectorized::DataTypePtr value_type);
+
+ // create partitions when need for auto-partition table using #_partitions_need_create.
+ Status _automatic_create_partition();
+
+ void _get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids);
+
+ void _filter_block_by_skip(vectorized::Block* block, RowPartTabletTuple& row_part_tablet_tuple);
+
+ Status _filter_block_by_skip_and_where_clause(vectorized::Block* block,
+ const vectorized::VExprContextSPtr& where_clause,
+ RowPartTabletTuple& row_part_tablet_tuple);
+
+ Status _filter_block(vectorized::Block* block,
+ std::vector<RowPartTabletTuple>& row_part_tablet_tuples);
+
+ Status _generate_rows_distribution_for_auto_parititon(
+ vectorized::Block* block, bool has_filtered_rows,
+ std::vector<RowPartTabletTuple>& row_part_tablet_tuples);
+
+ Status _generate_rows_distribution_for_non_auto_parititon(
+ vectorized::Block* block, bool has_filtered_rows,
+ std::vector<RowPartTabletTuple>& row_part_tablet_tuples);
+
+ void _reset_row_part_tablet_tuples(std::vector<RowPartTabletTuple>& row_part_tablet_tuples,
+ int64_t rows);
+
+private:
+ RuntimeState* _state = nullptr;
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/vec/sink/vrow_distribution.h:94:** previously declared here
```cpp
private:
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1797804696
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1795132348
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798276581
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.95 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162290654 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26432:
URL: https://github.com/apache/doris/pull/26432#discussion_r1384804918
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,306 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/sink/vrow_distribution.h"
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
+ vectorized::DataTypePtr value_type) {
+ _partitions_need_create.clear();
+ std::set<std::string> deduper;
+ // de-duplication
+ for (auto row : _missing_map) {
+ deduper.emplace(value_type->to_string(*col, row));
+ }
+ for (auto& value : deduper) {
+ TStringLiteral node;
+ node.value = value;
+ _partitions_need_create.emplace_back(std::vector {node}); // only 1 partition column now
+ }
+}
+
+Status VRowDistribution::_automatic_create_partition() {
+ SCOPED_TIMER(_add_partition_request_timer);
+ TCreatePartitionRequest request;
+ TCreatePartitionResult result;
+ request.__set_txn_id(_txn_id);
+ request.__set_db_id(_vpartition->db_id());
+ request.__set_table_id(_vpartition->table_id());
+ request.__set_partitionValues(_partitions_need_create);
+
+ VLOG(1) << "automatic partition rpc begin request " << request;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ int time_out = _state->execution_timeout() * 1000;
+ RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
+ master_addr.hostname, master_addr.port,
+ [&request, &result](FrontendServiceConnection& client) {
+ client->createPartition(result, request);
+ },
+ time_out));
+
+ Status status(Status::create(result.status));
+ VLOG(1) << "automatic partition rpc end response " << result;
+ if (result.status.status_code == TStatusCode::OK) {
+ // add new created partitions
+ RETURN_IF_ERROR(_vpartition->add_partitions(result.partitions));
+ RETURN_IF_ERROR(_on_partitions_created(_caller, &result));
+ }
+
+ return status;
+}
+
+void VRowDistribution::_get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids) {
+ tablet_ids.reserve(block->rows());
+ for (int row_idx = 0; row_idx < block->rows(); row_idx++) {
+ if (_skip[row_idx]) {
+ continue;
+ }
+ auto& partition = _partitions[row_idx];
+ auto& tablet_index = _tablet_indexes[row_idx];
+ auto& index = partition->indexes[index_idx];
+
+ auto tablet_id = index.tablets[tablet_index];
+ tablet_ids[row_idx] = tablet_id;
+ }
+}
+
+void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
Review Comment:
warning: method '_filter_block_by_skip' can be made static [readability-convert-member-functions-to-static]
be/src/vec/sink/vrow_distribution.h:110:
```diff
- void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_id);
+ static void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798826866
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1797280841
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1800885540
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1793746600
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.28 seconds
stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 55.8 seconds inserted 10000000 Rows, about 179K ops/s
storage size: 17162303039 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1794972927
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798977285
TeamCity be ut coverage result:
Function Coverage: 37.04% (8403/22687)
Line Coverage: 29.47% (68030/230840)
Region Coverage: 28.14% (35164/124980)
Branch Coverage: 24.93% (17975/72102)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f41908fdea89bf8e28fb06a63c71ff732ed58f2f_f41908fdea89bf8e28fb06a63c71ff732ed58f2f/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798322677
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798353691
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26432:
URL: https://github.com/apache/doris/pull/26432#discussion_r1382537027
##########
be/src/exec/tablet_info.h:
##########
@@ -17,6 +17,7 @@
#pragma once
+#include <butil/fast_rand.h>
Review Comment:
warning: 'butil/fast_rand.h' file not found [clang-diagnostic-error]
```cpp
#include <butil/fast_rand.h>
^
```
##########
be/src/exec/tablet_info.h:
##########
@@ -162,9 +165,78 @@
int64_t version() const { return _t_param.version; }
// return true if we found this block_row in partition
- bool find_partition(BlockRow* block_row, const VOlapTablePartition** partition) const;
+ //TODO: use virtual function to refactor it
+ ALWAYS_INLINE bool find_partition(vectorized::Block* block, int row,
+ VOlapTablePartition*& partition) const {
+ auto it = _is_in_partition ? _partitions_map->find(std::tuple {block, row, true})
+ : _partitions_map->upper_bound(std::tuple {block, row, true});
+ // for list partition it might result in default partition
+ if (_is_in_partition) {
+ partition = (it != _partitions_map->end()) ? it->second : _default_partition;
+ it = _partitions_map->end();
+ }
+ if (it != _partitions_map->end() &&
+ _part_contains(it->second, std::tuple {block, row, true})) {
+ partition = it->second;
+ }
+ return (partition != nullptr);
+ }
+
+ ALWAYS_INLINE void find_tablets(
Review Comment:
warning: method 'find_tablets' can be made static [readability-convert-member-functions-to-static]
```suggestion
static ALWAYS_INLINE void find_tablets(
```
be/src/exec/tablet_info.h:189:
```diff
- std::map<int64_t, int64_t>* partition_tablets_buffer = nullptr) const {
+ std::map<int64_t, int64_t>* partition_tablets_buffer = nullptr) {
```
##########
be/src/vec/sink/vtablet_sink_v2.cpp:
##########
@@ -221,20 +221,32 @@ void VOlapTableSinkV2::_build_tablet_node_mapping() {
}
}
-void VOlapTableSinkV2::_generate_rows_for_tablet(RowsForTablet& rows_for_tablet,
- const VOlapTablePartition* partition,
- uint32_t tablet_index, int row_idx) {
- // Generate channel payload for sinking data to each tablet
- for (const auto& index : partition->indexes) {
- auto tablet_id = index.tablets[tablet_index];
- if (rows_for_tablet.count(tablet_id) == 0) {
- Rows rows;
- rows.partition_id = partition->id;
- rows.index_id = index.index_id;
- rows_for_tablet.insert({tablet_id, rows});
+void VOlapTableSinkV2::_generate_rows_for_tablet(
Review Comment:
warning: method '_generate_rows_for_tablet' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void VOlapTableSinkV2::_generate_rows_for_tablet(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #26432:
URL: https://github.com/apache/doris/pull/26432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Row distribution [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26432:
URL: https://github.com/apache/doris/pull/26432#discussion_r1383323741
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,288 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/sink/vrow_distribution.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
+ vectorized::DataTypePtr value_type) {
+ _partitions_need_create.clear();
+ std::set<std::string> deduper;
+ // de-duplication
+ for (auto row : _missing_map) {
+ deduper.emplace(value_type->to_string(*col, row));
+ }
+ for (auto& value : deduper) {
+ TStringLiteral node;
+ node.value = value;
+ _partitions_need_create.emplace_back(std::vector {node}); // only 1 partition column now
+ }
+}
+
+Status VRowDistribution::_automatic_create_partition() {
+ SCOPED_TIMER(_add_partition_request_timer);
+ TCreatePartitionRequest request;
+ TCreatePartitionResult result;
+ request.__set_txn_id(_txn_id);
+ request.__set_db_id(_vpartition->db_id());
+ request.__set_table_id(_vpartition->table_id());
+ request.__set_partitionValues(_partitions_need_create);
+
+ VLOG(1) << "automatic partition rpc begin request " << request;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ int time_out = _state->execution_timeout() * 1000;
+ RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
+ master_addr.hostname, master_addr.port,
+ [&request, &result](FrontendServiceConnection& client) {
+ client->createPartition(result, request);
+ },
+ time_out));
+
+ Status status(Status::create(result.status));
+ VLOG(1) << "automatic partition rpc end response " << result;
+ if (result.status.status_code == TStatusCode::OK) {
+ // add new created partitions
+ RETURN_IF_ERROR(_vpartition->add_partitions(result.partitions));
+ RETURN_IF_ERROR(_on_partitions_created(_caller, &result));
+ }
+
+ return status;
+}
+
+void VRowDistribution::_get_tablet_ids(vectorized::Block* block, int32_t index_idx,
Review Comment:
warning: method '_get_tablet_ids' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void VRowDistribution::_get_tablet_ids(vectorized::Block* block, int32_t index_idx,
```
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,288 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/sink/vrow_distribution.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
+ vectorized::DataTypePtr value_type) {
+ _partitions_need_create.clear();
+ std::set<std::string> deduper;
+ // de-duplication
+ for (auto row : _missing_map) {
+ deduper.emplace(value_type->to_string(*col, row));
+ }
+ for (auto& value : deduper) {
+ TStringLiteral node;
+ node.value = value;
+ _partitions_need_create.emplace_back(std::vector {node}); // only 1 partition column now
+ }
+}
+
+Status VRowDistribution::_automatic_create_partition() {
+ SCOPED_TIMER(_add_partition_request_timer);
+ TCreatePartitionRequest request;
+ TCreatePartitionResult result;
+ request.__set_txn_id(_txn_id);
+ request.__set_db_id(_vpartition->db_id());
+ request.__set_table_id(_vpartition->table_id());
+ request.__set_partitionValues(_partitions_need_create);
+
+ VLOG(1) << "automatic partition rpc begin request " << request;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ int time_out = _state->execution_timeout() * 1000;
+ RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
+ master_addr.hostname, master_addr.port,
+ [&request, &result](FrontendServiceConnection& client) {
+ client->createPartition(result, request);
+ },
+ time_out));
+
+ Status status(Status::create(result.status));
+ VLOG(1) << "automatic partition rpc end response " << result;
+ if (result.status.status_code == TStatusCode::OK) {
+ // add new created partitions
+ RETURN_IF_ERROR(_vpartition->add_partitions(result.partitions));
+ RETURN_IF_ERROR(_on_partitions_created(_caller, &result));
+ }
+
+ return status;
+}
+
+void VRowDistribution::_get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids) {
+ tablet_ids.reserve(block->rows());
+ for (int row_idx = 0; row_idx < block->rows(); row_idx++) {
+ if (_skip[row_idx]) {
+ continue;
+ }
+ auto& partition = _partitions[row_idx];
+ auto& tablet_index = _tablet_indexes[row_idx];
+ auto& index = partition->indexes[index_idx];
+
+ auto tablet_id = index.tablets[tablet_index];
+ tablet_ids[row_idx] = tablet_id;
+ }
+}
+
+void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
Review Comment:
warning: method '_filter_block_by_skip' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+using RowPartTabletTuple = std::tuple<std::vector<int64_t>, std::vector<int64_t>,
+ std::vector<int64_t>>;
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr; // not owned, set when open
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {
+ }
+
+ void init(VRowDistributionContext *ctx) {
+ _state = ctx->state;
+ _channels = ctx->channels;
+ _block_convertor = ctx->block_convertor;
+ _tablet_finder = ctx->tablet_finder;
+ _vpartition = ctx->vpartition;
+ _add_partition_request_timer = ctx->add_partition_request_timer;
+ _txn_id = ctx->txn_id;
+ _pool = ctx->pool;
+ _location = ctx->location;
+ _vec_output_expr_ctxs = ctx->vec_output_expr_ctxs;
+ _on_partitions_created = ctx->on_partitions_created;
+ _caller = ctx->caller;
+ _schema = ctx->schema;
+ }
+
+ // auto partition
+ // mv where clause
+ // v1 needs index->node->row_ids - tabletids
+ // v2 needs index,tablet->rowids
+ Status generate_rows_distribution(vectorized::Block& input_block,
+ std::shared_ptr<vectorized::Block>& block,
+ int64_t& filtered_rows, bool& has_filtered_rows,
+ std::vector<RowPartTabletTuple>& row_part_tablet_tuples);
+
+private:
+ std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr> _get_partition_function();
+
+ void _save_missing_values(vectorized::ColumnPtr col, vectorized::DataTypePtr value_type);
+
+ // create partitions when need for auto-partition table using #_partitions_need_create.
+ Status _automatic_create_partition();
+
+ void _get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids);
+
+ void _filter_block_by_skip(vectorized::Block* block,
+ RowPartTabletTuple& row_part_tablet_tuple);
+
+ Status _filter_block_by_skip_and_where_clause(vectorized::Block* block,
+ const vectorized::VExprContextSPtr& where_clause, RowPartTabletTuple& row_part_tablet_tuple);
+
+ Status _filter_block(vectorized::Block* block, std::vector<RowPartTabletTuple>& row_part_tablet_tuples);
+
+ Status _generate_rows_distribution_for_auto_parititon(
+ vectorized::Block* block, bool has_filtered_rows,
+ std::vector<RowPartTabletTuple>& row_part_tablet_tuples);
+
+ Status _generate_rows_distribution_for_non_auto_parititon(
+ vectorized::Block* block, bool has_filtered_rows,
+ std::vector<RowPartTabletTuple>& row_part_tablet_tuples);
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/vec/sink/vrow_distribution.h:93:** previously declared here
```cpp
private:
^
```
</details>
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,288 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/sink/vrow_distribution.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
+ vectorized::DataTypePtr value_type) {
+ _partitions_need_create.clear();
+ std::set<std::string> deduper;
+ // de-duplication
+ for (auto row : _missing_map) {
+ deduper.emplace(value_type->to_string(*col, row));
+ }
+ for (auto& value : deduper) {
+ TStringLiteral node;
+ node.value = value;
+ _partitions_need_create.emplace_back(std::vector {node}); // only 1 partition column now
+ }
+}
+
+Status VRowDistribution::_automatic_create_partition() {
+ SCOPED_TIMER(_add_partition_request_timer);
+ TCreatePartitionRequest request;
+ TCreatePartitionResult result;
+ request.__set_txn_id(_txn_id);
+ request.__set_db_id(_vpartition->db_id());
+ request.__set_table_id(_vpartition->table_id());
+ request.__set_partitionValues(_partitions_need_create);
+
+ VLOG(1) << "automatic partition rpc begin request " << request;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ int time_out = _state->execution_timeout() * 1000;
+ RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
+ master_addr.hostname, master_addr.port,
+ [&request, &result](FrontendServiceConnection& client) {
+ client->createPartition(result, request);
+ },
+ time_out));
+
+ Status status(Status::create(result.status));
+ VLOG(1) << "automatic partition rpc end response " << result;
+ if (result.status.status_code == TStatusCode::OK) {
+ // add new created partitions
+ RETURN_IF_ERROR(_vpartition->add_partitions(result.partitions));
+ RETURN_IF_ERROR(_on_partitions_created(_caller, &result));
+ }
+
+ return status;
+}
+
+void VRowDistribution::_get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids) {
+ tablet_ids.reserve(block->rows());
+ for (int row_idx = 0; row_idx < block->rows(); row_idx++) {
+ if (_skip[row_idx]) {
+ continue;
+ }
+ auto& partition = _partitions[row_idx];
+ auto& tablet_index = _tablet_indexes[row_idx];
+ auto& index = partition->indexes[index_idx];
+
+ auto tablet_id = index.tablets[tablet_index];
+ tablet_ids[row_idx] = tablet_id;
+ }
+}
+
+void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
+ RowPartTabletTuple& row_part_tablet_tuple) {
+ auto& row_ids = std::get<0>(row_part_tablet_tuple);
+ auto& partition_ids = std::get<1>(row_part_tablet_tuple);
+ auto& tablet_ids = std::get<2>(row_part_tablet_tuple);
+
+ for (size_t i = 0; i < block->rows(); i++) {
+ if (!_skip[i]) {
+ row_ids.emplace_back(i);
+ partition_ids.emplace_back(_partitions[i]->id);
+ tablet_ids.emplace_back(_tablet_ids[i]);
+ }
+ }
+}
+
+Status VRowDistribution::_filter_block_by_skip_and_where_clause(vectorized::Block* block,
Review Comment:
warning: method '_filter_block_by_skip_and_where_clause' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status VRowDistribution::_filter_block_by_skip_and_where_clause(vectorized::Block* block,
```
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,288 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/sink/vrow_distribution.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
+ vectorized::DataTypePtr value_type) {
+ _partitions_need_create.clear();
+ std::set<std::string> deduper;
+ // de-duplication
+ for (auto row : _missing_map) {
+ deduper.emplace(value_type->to_string(*col, row));
+ }
+ for (auto& value : deduper) {
+ TStringLiteral node;
+ node.value = value;
+ _partitions_need_create.emplace_back(std::vector {node}); // only 1 partition column now
+ }
+}
+
+Status VRowDistribution::_automatic_create_partition() {
+ SCOPED_TIMER(_add_partition_request_timer);
+ TCreatePartitionRequest request;
+ TCreatePartitionResult result;
+ request.__set_txn_id(_txn_id);
+ request.__set_db_id(_vpartition->db_id());
+ request.__set_table_id(_vpartition->table_id());
+ request.__set_partitionValues(_partitions_need_create);
+
+ VLOG(1) << "automatic partition rpc begin request " << request;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ int time_out = _state->execution_timeout() * 1000;
+ RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
+ master_addr.hostname, master_addr.port,
+ [&request, &result](FrontendServiceConnection& client) {
+ client->createPartition(result, request);
+ },
+ time_out));
+
+ Status status(Status::create(result.status));
+ VLOG(1) << "automatic partition rpc end response " << result;
+ if (result.status.status_code == TStatusCode::OK) {
+ // add new created partitions
+ RETURN_IF_ERROR(_vpartition->add_partitions(result.partitions));
+ RETURN_IF_ERROR(_on_partitions_created(_caller, &result));
+ }
+
+ return status;
+}
+
+void VRowDistribution::_get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids) {
+ tablet_ids.reserve(block->rows());
+ for (int row_idx = 0; row_idx < block->rows(); row_idx++) {
+ if (_skip[row_idx]) {
+ continue;
+ }
+ auto& partition = _partitions[row_idx];
+ auto& tablet_index = _tablet_indexes[row_idx];
+ auto& index = partition->indexes[index_idx];
+
+ auto tablet_id = index.tablets[tablet_index];
+ tablet_ids[row_idx] = tablet_id;
+ }
+}
+
+void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
+ RowPartTabletTuple& row_part_tablet_tuple) {
+ auto& row_ids = std::get<0>(row_part_tablet_tuple);
+ auto& partition_ids = std::get<1>(row_part_tablet_tuple);
+ auto& tablet_ids = std::get<2>(row_part_tablet_tuple);
+
+ for (size_t i = 0; i < block->rows(); i++) {
+ if (!_skip[i]) {
+ row_ids.emplace_back(i);
+ partition_ids.emplace_back(_partitions[i]->id);
+ tablet_ids.emplace_back(_tablet_ids[i]);
+ }
+ }
+}
+
+Status VRowDistribution::_filter_block_by_skip_and_where_clause(vectorized::Block* block,
+ const vectorized::VExprContextSPtr& where_clause, RowPartTabletTuple& row_part_tablet_tuple) {
+
+ // TODO
+ //SCOPED_RAW_TIMER(&_stat.where_clause_ns);
+ int result_index = -1;
+ size_t column_number = block->columns();
+ RETURN_IF_ERROR(where_clause->execute(block, &result_index));
+
+ auto filter_column = block->get_by_position(result_index).column;
+
+ auto& row_ids = std::get<0>(row_part_tablet_tuple);
+ auto& partition_ids = std::get<1>(row_part_tablet_tuple);
+ auto& tablet_ids = std::get<2>(row_part_tablet_tuple);
+ if (auto* nullable_column =
+ vectorized::check_and_get_column<vectorized::ColumnNullable>(*filter_column)) {
+ for (size_t i = 0; i < block->rows(); i++) {
+ if (nullable_column->get_bool_inline(i) && !_skip[i]) {
+ row_ids.emplace_back(i);
+ partition_ids.emplace_back(_partitions[i]->id);
+ tablet_ids.emplace_back(_tablet_ids[i]);
+ }
+ }
+ } else if (auto* const_column = vectorized::check_and_get_column<vectorized::ColumnConst>(
+ *filter_column)) {
+ bool ret = const_column->get_bool(0);
+ if (!ret) {
+ return Status::OK();
+ }
+ // should we optimize?
+ _filter_block_by_skip(block, row_part_tablet_tuple);
+ } else {
+ auto& filter = assert_cast<const vectorized::ColumnUInt8&>(*filter_column).get_data();
+ for (size_t i = 0; i < block->rows(); i++) {
+ if (filter[i] != 0 && !_skip[i]) {
+ row_ids.emplace_back(i);
+ partition_ids.emplace_back(_partitions[i]->id);
+ tablet_ids.emplace_back(_tablet_ids[i]);
+ }
+ }
+ }
+
+ for (size_t i = block->columns() - 1; i >= column_number; i--) {
+ block->erase(i);
+ }
+ return Status::OK();
+}
+
+Status VRowDistribution::_filter_block(vectorized::Block* block,
+ std::vector<RowPartTabletTuple>& row_part_tablet_tuples) {
+ for (int i = 0; i < _schema->indexes().size(); i++) {
+ _get_tablet_ids(block, i, _tablet_ids);
+ auto& where_clause = _schema->indexes()[i]->where_clause;
+ if (where_clause != nullptr) {
+ RETURN_IF_ERROR(_filter_block_by_skip_and_where_clause(block, where_clause, row_part_tablet_tuples[i]));
+ } else {
+ _filter_block_by_skip(block, row_part_tablet_tuples[i]);
+ }
+ }
+ return Status::OK();
+}
+
+Status VRowDistribution::_generate_rows_distribution_for_non_auto_parititon(
Review Comment:
warning: method '_generate_rows_distribution_for_non_auto_parititon' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status VRowDistribution::_generate_rows_distribution_for_non_auto_parititon(
```
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,288 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/sink/vrow_distribution.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
+ vectorized::DataTypePtr value_type) {
+ _partitions_need_create.clear();
+ std::set<std::string> deduper;
+ // de-duplication
+ for (auto row : _missing_map) {
+ deduper.emplace(value_type->to_string(*col, row));
+ }
+ for (auto& value : deduper) {
+ TStringLiteral node;
+ node.value = value;
+ _partitions_need_create.emplace_back(std::vector {node}); // only 1 partition column now
+ }
+}
+
+Status VRowDistribution::_automatic_create_partition() {
+ SCOPED_TIMER(_add_partition_request_timer);
+ TCreatePartitionRequest request;
+ TCreatePartitionResult result;
+ request.__set_txn_id(_txn_id);
+ request.__set_db_id(_vpartition->db_id());
+ request.__set_table_id(_vpartition->table_id());
+ request.__set_partitionValues(_partitions_need_create);
+
+ VLOG(1) << "automatic partition rpc begin request " << request;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ int time_out = _state->execution_timeout() * 1000;
+ RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
+ master_addr.hostname, master_addr.port,
+ [&request, &result](FrontendServiceConnection& client) {
+ client->createPartition(result, request);
+ },
+ time_out));
+
+ Status status(Status::create(result.status));
+ VLOG(1) << "automatic partition rpc end response " << result;
+ if (result.status.status_code == TStatusCode::OK) {
+ // add new created partitions
+ RETURN_IF_ERROR(_vpartition->add_partitions(result.partitions));
+ RETURN_IF_ERROR(_on_partitions_created(_caller, &result));
+ }
+
+ return status;
+}
+
+void VRowDistribution::_get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids) {
+ tablet_ids.reserve(block->rows());
+ for (int row_idx = 0; row_idx < block->rows(); row_idx++) {
+ if (_skip[row_idx]) {
+ continue;
+ }
+ auto& partition = _partitions[row_idx];
+ auto& tablet_index = _tablet_indexes[row_idx];
+ auto& index = partition->indexes[index_idx];
+
+ auto tablet_id = index.tablets[tablet_index];
+ tablet_ids[row_idx] = tablet_id;
+ }
+}
+
+void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
+ RowPartTabletTuple& row_part_tablet_tuple) {
+ auto& row_ids = std::get<0>(row_part_tablet_tuple);
+ auto& partition_ids = std::get<1>(row_part_tablet_tuple);
+ auto& tablet_ids = std::get<2>(row_part_tablet_tuple);
+
+ for (size_t i = 0; i < block->rows(); i++) {
+ if (!_skip[i]) {
+ row_ids.emplace_back(i);
+ partition_ids.emplace_back(_partitions[i]->id);
+ tablet_ids.emplace_back(_tablet_ids[i]);
+ }
+ }
+}
+
+Status VRowDistribution::_filter_block_by_skip_and_where_clause(vectorized::Block* block,
+ const vectorized::VExprContextSPtr& where_clause, RowPartTabletTuple& row_part_tablet_tuple) {
+
+ // TODO
+ //SCOPED_RAW_TIMER(&_stat.where_clause_ns);
+ int result_index = -1;
+ size_t column_number = block->columns();
+ RETURN_IF_ERROR(where_clause->execute(block, &result_index));
+
+ auto filter_column = block->get_by_position(result_index).column;
+
+ auto& row_ids = std::get<0>(row_part_tablet_tuple);
+ auto& partition_ids = std::get<1>(row_part_tablet_tuple);
+ auto& tablet_ids = std::get<2>(row_part_tablet_tuple);
+ if (auto* nullable_column =
+ vectorized::check_and_get_column<vectorized::ColumnNullable>(*filter_column)) {
+ for (size_t i = 0; i < block->rows(); i++) {
+ if (nullable_column->get_bool_inline(i) && !_skip[i]) {
+ row_ids.emplace_back(i);
+ partition_ids.emplace_back(_partitions[i]->id);
+ tablet_ids.emplace_back(_tablet_ids[i]);
+ }
+ }
+ } else if (auto* const_column = vectorized::check_and_get_column<vectorized::ColumnConst>(
+ *filter_column)) {
+ bool ret = const_column->get_bool(0);
+ if (!ret) {
+ return Status::OK();
+ }
+ // should we optimize?
+ _filter_block_by_skip(block, row_part_tablet_tuple);
+ } else {
+ auto& filter = assert_cast<const vectorized::ColumnUInt8&>(*filter_column).get_data();
+ for (size_t i = 0; i < block->rows(); i++) {
+ if (filter[i] != 0 && !_skip[i]) {
+ row_ids.emplace_back(i);
+ partition_ids.emplace_back(_partitions[i]->id);
+ tablet_ids.emplace_back(_tablet_ids[i]);
+ }
+ }
+ }
+
+ for (size_t i = block->columns() - 1; i >= column_number; i--) {
+ block->erase(i);
+ }
+ return Status::OK();
+}
+
+Status VRowDistribution::_filter_block(vectorized::Block* block,
Review Comment:
warning: method '_filter_block' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status VRowDistribution::_filter_block(vectorized::Block* block,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26432:
URL: https://github.com/apache/doris/pull/26432#discussion_r1384491141
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,305 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/sink/vrow_distribution.h"
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
+ vectorized::DataTypePtr value_type) {
+ _partitions_need_create.clear();
+ std::set<std::string> deduper;
+ // de-duplication
+ for (auto row : _missing_map) {
+ deduper.emplace(value_type->to_string(*col, row));
+ }
+ for (auto& value : deduper) {
+ TStringLiteral node;
+ node.value = value;
+ _partitions_need_create.emplace_back(std::vector {node}); // only 1 partition column now
+ }
+}
+
+Status VRowDistribution::_automatic_create_partition() {
+ SCOPED_TIMER(_add_partition_request_timer);
+ TCreatePartitionRequest request;
+ TCreatePartitionResult result;
+ request.__set_txn_id(_txn_id);
+ request.__set_db_id(_vpartition->db_id());
+ request.__set_table_id(_vpartition->table_id());
+ request.__set_partitionValues(_partitions_need_create);
+
+ VLOG(1) << "automatic partition rpc begin request " << request;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ int time_out = _state->execution_timeout() * 1000;
+ RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
+ master_addr.hostname, master_addr.port,
+ [&request, &result](FrontendServiceConnection& client) {
+ client->createPartition(result, request);
+ },
+ time_out));
+
+ Status status(Status::create(result.status));
+ VLOG(1) << "automatic partition rpc end response " << result;
+ if (result.status.status_code == TStatusCode::OK) {
+ // add new created partitions
+ RETURN_IF_ERROR(_vpartition->add_partitions(result.partitions));
+ RETURN_IF_ERROR(_on_partitions_created(_caller, &result));
+ }
+
+ return status;
+}
+
+void VRowDistribution::_get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids) {
+ tablet_ids.reserve(block->rows());
+ for (int row_idx = 0; row_idx < block->rows(); row_idx++) {
+ if (_skip[row_idx]) {
+ continue;
+ }
+ auto& partition = _partitions[row_idx];
+ auto& tablet_index = _tablet_indexes[row_idx];
+ auto& index = partition->indexes[index_idx];
+
+ auto tablet_id = index.tablets[tablet_index];
+ tablet_ids[row_idx] = tablet_id;
+ }
+}
+
+void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
Review Comment:
warning: method '_filter_block_by_skip' can be made static [readability-convert-member-functions-to-static]
be/src/vec/sink/vrow_distribution.h:112:
```diff
- void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
+ static void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include <gen_cpp/FrontendService.h>
Review Comment:
warning: 'gen_cpp/FrontendService.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/FrontendService.h>
^
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/PaloInternalService_types.h>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+class RowPartTabletIds {
+public:
+ std::vector<int64_t> row_ids;
+ std::vector<int64_t> partition_ids;
+ std::vector<int64_t> tablet_ids;
+};
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
+ virtual ~VRowDistribution() {}
+
+ void init(VRowDistributionContext* ctx) {
+ _state = ctx->state;
+ _channels = ctx->channels;
+ _block_convertor = ctx->block_convertor;
+ _tablet_finder = ctx->tablet_finder;
+ _vpartition = ctx->vpartition;
+ _add_partition_request_timer = ctx->add_partition_request_timer;
+ _txn_id = ctx->txn_id;
+ _pool = ctx->pool;
+ _location = ctx->location;
+ _vec_output_expr_ctxs = ctx->vec_output_expr_ctxs;
+ _on_partitions_created = ctx->on_partitions_created;
+ _caller = ctx->caller;
+ _schema = ctx->schema;
+ }
+
+ // auto partition
+ // mv where clause
+ // v1 needs index->node->row_ids - tabletids
+ // v2 needs index,tablet->rowids
+ Status generate_rows_distribution(vectorized::Block& input_block,
+ std::shared_ptr<vectorized::Block>& block,
+ int64_t& filtered_rows, bool& has_filtered_rows,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+private:
+ std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr> _get_partition_function();
+
+ void _save_missing_values(vectorized::ColumnPtr col, vectorized::DataTypePtr value_type);
+
+ // create partitions when need for auto-partition table using #_partitions_need_create.
+ Status _automatic_create_partition();
+
+ void _get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids);
+
+ void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
Review Comment:
warning: function 'std::doris::vectorized::VRowDistribution::_filter_block_by_skip' has a definition with different parameter names [readability-inconsistent-declaration-parameter-name]
```cpp
void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
^
```
<details>
<summary>Additional context</summary>
**be/src/vec/sink/vrow_distribution.cpp:99:** the definition seen here
```cpp
void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
^
```
**be/src/vec/sink/vrow_distribution.h:112:** differing parameters are named here: ('row_part_tablet_tuple'), in definition: ('row_part_tablet_id')
```cpp
void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
^
```
</details>
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/PaloInternalService_types.h>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+class RowPartTabletIds {
+public:
+ std::vector<int64_t> row_ids;
+ std::vector<int64_t> partition_ids;
+ std::vector<int64_t> tablet_ids;
+};
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
+ virtual ~VRowDistribution() {}
+
+ void init(VRowDistributionContext* ctx) {
+ _state = ctx->state;
+ _channels = ctx->channels;
+ _block_convertor = ctx->block_convertor;
+ _tablet_finder = ctx->tablet_finder;
+ _vpartition = ctx->vpartition;
+ _add_partition_request_timer = ctx->add_partition_request_timer;
+ _txn_id = ctx->txn_id;
+ _pool = ctx->pool;
+ _location = ctx->location;
+ _vec_output_expr_ctxs = ctx->vec_output_expr_ctxs;
+ _on_partitions_created = ctx->on_partitions_created;
+ _caller = ctx->caller;
+ _schema = ctx->schema;
+ }
+
+ // auto partition
+ // mv where clause
+ // v1 needs index->node->row_ids - tabletids
+ // v2 needs index,tablet->rowids
+ Status generate_rows_distribution(vectorized::Block& input_block,
+ std::shared_ptr<vectorized::Block>& block,
+ int64_t& filtered_rows, bool& has_filtered_rows,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+private:
+ std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr> _get_partition_function();
+
+ void _save_missing_values(vectorized::ColumnPtr col, vectorized::DataTypePtr value_type);
+
+ // create partitions when need for auto-partition table using #_partitions_need_create.
+ Status _automatic_create_partition();
+
+ void _get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids);
+
+ void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
+
+ Status _filter_block_by_skip_and_where_clause(vectorized::Block* block,
+ const vectorized::VExprContextSPtr& where_clause,
+ RowPartTabletIds& row_part_tablet_tuple);
+
+ Status _filter_block(vectorized::Block* block,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+ Status _generate_rows_distribution_for_auto_parititon(
+ vectorized::Block* block, bool has_filtered_rows,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+ Status _generate_rows_distribution_for_non_auto_parititon(
+ vectorized::Block* block, bool has_filtered_rows,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+ void _reset_row_part_tablet_ids(std::vector<RowPartTabletIds>& row_part_tablet_ids,
+ int64_t rows);
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/vec/sink/vrow_distribution.h:101:** previously declared here
```cpp
private:
^
```
</details>
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/PaloInternalService_types.h>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+class RowPartTabletIds {
+public:
+ std::vector<int64_t> row_ids;
+ std::vector<int64_t> partition_ids;
+ std::vector<int64_t> tablet_ids;
+};
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ std::vector<std::shared_ptr<IndexChannel>>* channels;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
+ virtual ~VRowDistribution() {}
+
+ void init(VRowDistributionContext* ctx) {
+ _state = ctx->state;
+ _channels = ctx->channels;
+ _block_convertor = ctx->block_convertor;
+ _tablet_finder = ctx->tablet_finder;
+ _vpartition = ctx->vpartition;
+ _add_partition_request_timer = ctx->add_partition_request_timer;
+ _txn_id = ctx->txn_id;
+ _pool = ctx->pool;
+ _location = ctx->location;
+ _vec_output_expr_ctxs = ctx->vec_output_expr_ctxs;
+ _on_partitions_created = ctx->on_partitions_created;
+ _caller = ctx->caller;
+ _schema = ctx->schema;
+ }
+
+ // auto partition
+ // mv where clause
+ // v1 needs index->node->row_ids - tabletids
+ // v2 needs index,tablet->rowids
+ Status generate_rows_distribution(vectorized::Block& input_block,
+ std::shared_ptr<vectorized::Block>& block,
+ int64_t& filtered_rows, bool& has_filtered_rows,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+private:
+ std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr> _get_partition_function();
+
+ void _save_missing_values(vectorized::ColumnPtr col, vectorized::DataTypePtr value_type);
+
+ // create partitions when need for auto-partition table using #_partitions_need_create.
+ Status _automatic_create_partition();
+
+ void _get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids);
+
+ void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
+
+ Status _filter_block_by_skip_and_where_clause(vectorized::Block* block,
Review Comment:
warning: function 'std::doris::vectorized::VRowDistribution::_filter_block_by_skip_and_where_clause' has a definition with different parameter names [readability-inconsistent-declaration-parameter-name]
```cpp
Status _filter_block_by_skip_and_where_clause(vectorized::Block* block,
^
```
<details>
<summary>Additional context</summary>
**be/src/vec/sink/vrow_distribution.cpp:114:** the definition seen here
```cpp
Status VRowDistribution::_filter_block_by_skip_and_where_clause(
^
```
**be/src/vec/sink/vrow_distribution.h:114:** differing parameters are named here: ('row_part_tablet_tuple'), in definition: ('row_part_tablet_id')
```cpp
Status _filter_block_by_skip_and_where_clause(vectorized::Block* block,
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798015384
TeamCity be ut coverage result:
Function Coverage: 36.98% (8396/22702)
Line Coverage: 29.44% (68000/230973)
Region Coverage: 28.10% (35153/125085)
Branch Coverage: 24.90% (17967/72152)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e377110ad691a7ee5d892d542565eed164654bbf_e377110ad691a7ee5d892d542565eed164654bbf/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26432:
URL: https://github.com/apache/doris/pull/26432#discussion_r1384710662
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -0,0 +1,306 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/sink/vrow_distribution.h"
+
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
+#include "util/thrift_rpc_helper.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/sink/writer/vtablet_writer.h"
+
+namespace doris::vectorized {
+
+std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr>
+VRowDistribution::_get_partition_function() {
+ return {_vpartition->get_part_func_ctx(), _vpartition->get_partition_function()};
+}
+
+void VRowDistribution::_save_missing_values(vectorized::ColumnPtr col,
+ vectorized::DataTypePtr value_type) {
+ _partitions_need_create.clear();
+ std::set<std::string> deduper;
+ // de-duplication
+ for (auto row : _missing_map) {
+ deduper.emplace(value_type->to_string(*col, row));
+ }
+ for (auto& value : deduper) {
+ TStringLiteral node;
+ node.value = value;
+ _partitions_need_create.emplace_back(std::vector {node}); // only 1 partition column now
+ }
+}
+
+Status VRowDistribution::_automatic_create_partition() {
+ SCOPED_TIMER(_add_partition_request_timer);
+ TCreatePartitionRequest request;
+ TCreatePartitionResult result;
+ request.__set_txn_id(_txn_id);
+ request.__set_db_id(_vpartition->db_id());
+ request.__set_table_id(_vpartition->table_id());
+ request.__set_partitionValues(_partitions_need_create);
+
+ VLOG(1) << "automatic partition rpc begin request " << request;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ int time_out = _state->execution_timeout() * 1000;
+ RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
+ master_addr.hostname, master_addr.port,
+ [&request, &result](FrontendServiceConnection& client) {
+ client->createPartition(result, request);
+ },
+ time_out));
+
+ Status status(Status::create(result.status));
+ VLOG(1) << "automatic partition rpc end response " << result;
+ if (result.status.status_code == TStatusCode::OK) {
+ // add new created partitions
+ RETURN_IF_ERROR(_vpartition->add_partitions(result.partitions));
+ RETURN_IF_ERROR(_on_partitions_created(_caller, &result));
+ }
+
+ return status;
+}
+
+void VRowDistribution::_get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids) {
+ tablet_ids.reserve(block->rows());
+ for (int row_idx = 0; row_idx < block->rows(); row_idx++) {
+ if (_skip[row_idx]) {
+ continue;
+ }
+ auto& partition = _partitions[row_idx];
+ auto& tablet_index = _tablet_indexes[row_idx];
+ auto& index = partition->indexes[index_idx];
+
+ auto tablet_id = index.tablets[tablet_index];
+ tablet_ids[row_idx] = tablet_id;
+ }
+}
+
+void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
Review Comment:
warning: method '_filter_block_by_skip' can be made static [readability-convert-member-functions-to-static]
be/src/vec/sink/vrow_distribution.h:110:
```diff
- void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
+ static void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
```
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,160 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/PaloInternalService_types.h>
+
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+class RowPartTabletIds {
+public:
+ std::vector<int64_t> row_ids;
+ std::vector<int64_t> partition_ids;
+ std::vector<int64_t> tablet_ids;
+};
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
+ virtual ~VRowDistribution() {}
+
+ void init(VRowDistributionContext* ctx) {
+ _state = ctx->state;
+ _block_convertor = ctx->block_convertor;
+ _tablet_finder = ctx->tablet_finder;
+ _vpartition = ctx->vpartition;
+ _add_partition_request_timer = ctx->add_partition_request_timer;
+ _txn_id = ctx->txn_id;
+ _pool = ctx->pool;
+ _location = ctx->location;
+ _vec_output_expr_ctxs = ctx->vec_output_expr_ctxs;
+ _on_partitions_created = ctx->on_partitions_created;
+ _caller = ctx->caller;
+ _schema = ctx->schema;
+ }
+
+ // auto partition
+ // mv where clause
+ // v1 needs index->node->row_ids - tabletids
+ // v2 needs index,tablet->rowids
+ Status generate_rows_distribution(vectorized::Block& input_block,
+ std::shared_ptr<vectorized::Block>& block,
+ int64_t& filtered_rows, bool& has_filtered_rows,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+private:
+ std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr> _get_partition_function();
+
+ void _save_missing_values(vectorized::ColumnPtr col, vectorized::DataTypePtr value_type);
+
+ // create partitions when need for auto-partition table using #_partitions_need_create.
+ Status _automatic_create_partition();
+
+ void _get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids);
+
+ void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
Review Comment:
warning: function 'std::doris::vectorized::VRowDistribution::_filter_block_by_skip' has a definition with different parameter names [readability-inconsistent-declaration-parameter-name]
```cpp
void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
^
```
<details>
<summary>Additional context</summary>
**be/src/vec/sink/vrow_distribution.cpp:99:** the definition seen here
```cpp
void VRowDistribution::_filter_block_by_skip(vectorized::Block* block,
^
```
**be/src/vec/sink/vrow_distribution.h:110:** differing parameters are named here: ('row_part_tablet_tuple'), in definition: ('row_part_tablet_id')
```cpp
void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
^
```
</details>
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,160 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/PaloInternalService_types.h>
+
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+class RowPartTabletIds {
+public:
+ std::vector<int64_t> row_ids;
+ std::vector<int64_t> partition_ids;
+ std::vector<int64_t> tablet_ids;
+};
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
+ virtual ~VRowDistribution() {}
+
+ void init(VRowDistributionContext* ctx) {
+ _state = ctx->state;
+ _block_convertor = ctx->block_convertor;
+ _tablet_finder = ctx->tablet_finder;
+ _vpartition = ctx->vpartition;
+ _add_partition_request_timer = ctx->add_partition_request_timer;
+ _txn_id = ctx->txn_id;
+ _pool = ctx->pool;
+ _location = ctx->location;
+ _vec_output_expr_ctxs = ctx->vec_output_expr_ctxs;
+ _on_partitions_created = ctx->on_partitions_created;
+ _caller = ctx->caller;
+ _schema = ctx->schema;
+ }
+
+ // auto partition
+ // mv where clause
+ // v1 needs index->node->row_ids - tabletids
+ // v2 needs index,tablet->rowids
+ Status generate_rows_distribution(vectorized::Block& input_block,
+ std::shared_ptr<vectorized::Block>& block,
+ int64_t& filtered_rows, bool& has_filtered_rows,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+private:
+ std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr> _get_partition_function();
+
+ void _save_missing_values(vectorized::ColumnPtr col, vectorized::DataTypePtr value_type);
+
+ // create partitions when need for auto-partition table using #_partitions_need_create.
+ Status _automatic_create_partition();
+
+ void _get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids);
+
+ void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
+
+ Status _filter_block_by_skip_and_where_clause(vectorized::Block* block,
+ const vectorized::VExprContextSPtr& where_clause,
+ RowPartTabletIds& row_part_tablet_tuple);
+
+ Status _filter_block(vectorized::Block* block,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+ Status _generate_rows_distribution_for_auto_parititon(
+ vectorized::Block* block, bool has_filtered_rows,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+ Status _generate_rows_distribution_for_non_auto_parititon(
+ vectorized::Block* block, bool has_filtered_rows,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+ void _reset_row_part_tablet_ids(std::vector<RowPartTabletIds>& row_part_tablet_ids,
+ int64_t rows);
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/vec/sink/vrow_distribution.h:99:** previously declared here
```cpp
private:
^
```
</details>
##########
be/src/vec/sink/vrow_distribution.h:
##########
@@ -0,0 +1,160 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// IWYU pragma: no_include <bits/chrono.h>
+#include <gen_cpp/FrontendService.h>
+#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/PaloInternalService_types.h>
+
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "common/status.h"
+#include "exec/tablet_info.h"
+#include "runtime/types.h"
+#include "util/runtime_profile.h"
+#include "util/stopwatch.hpp"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr_fwd.h"
+#include "vec/sink/vtablet_block_convertor.h"
+#include "vec/sink/vtablet_finder.h"
+
+namespace doris::vectorized {
+
+class IndexChannel;
+class VNodeChannel;
+
+// <row_idx, partition_id, tablet_id>
+class RowPartTabletIds {
+public:
+ std::vector<int64_t> row_ids;
+ std::vector<int64_t> partition_ids;
+ std::vector<int64_t> tablet_ids;
+};
+
+typedef Status (*OnPartitionsCreated)(void*, TCreatePartitionResult*);
+
+class VRowDistributionContext {
+public:
+ RuntimeState* state = nullptr;
+ OlapTableBlockConvertor* block_convertor = nullptr;
+ OlapTabletFinder* tablet_finder = nullptr;
+ VOlapTablePartitionParam* vpartition = nullptr;
+ RuntimeProfile::Counter* add_partition_request_timer = nullptr;
+ int64_t txn_id = -1;
+ ObjectPool* pool;
+ OlapTableLocationParam* location;
+ const VExprContextSPtrs* vec_output_expr_ctxs;
+ OnPartitionsCreated on_partitions_created;
+ void* caller;
+ std::shared_ptr<OlapTableSchemaParam> schema;
+};
+
+class VRowDistribution {
+public:
+ VRowDistribution() {}
+ virtual ~VRowDistribution() {}
+
+ void init(VRowDistributionContext* ctx) {
+ _state = ctx->state;
+ _block_convertor = ctx->block_convertor;
+ _tablet_finder = ctx->tablet_finder;
+ _vpartition = ctx->vpartition;
+ _add_partition_request_timer = ctx->add_partition_request_timer;
+ _txn_id = ctx->txn_id;
+ _pool = ctx->pool;
+ _location = ctx->location;
+ _vec_output_expr_ctxs = ctx->vec_output_expr_ctxs;
+ _on_partitions_created = ctx->on_partitions_created;
+ _caller = ctx->caller;
+ _schema = ctx->schema;
+ }
+
+ // auto partition
+ // mv where clause
+ // v1 needs index->node->row_ids - tabletids
+ // v2 needs index,tablet->rowids
+ Status generate_rows_distribution(vectorized::Block& input_block,
+ std::shared_ptr<vectorized::Block>& block,
+ int64_t& filtered_rows, bool& has_filtered_rows,
+ std::vector<RowPartTabletIds>& row_part_tablet_ids);
+
+private:
+ std::pair<vectorized::VExprContextSPtr, vectorized::VExprSPtr> _get_partition_function();
+
+ void _save_missing_values(vectorized::ColumnPtr col, vectorized::DataTypePtr value_type);
+
+ // create partitions when need for auto-partition table using #_partitions_need_create.
+ Status _automatic_create_partition();
+
+ void _get_tablet_ids(vectorized::Block* block, int32_t index_idx,
+ std::vector<int64_t>& tablet_ids);
+
+ void _filter_block_by_skip(vectorized::Block* block, RowPartTabletIds& row_part_tablet_tuple);
+
+ Status _filter_block_by_skip_and_where_clause(vectorized::Block* block,
Review Comment:
warning: function 'std::doris::vectorized::VRowDistribution::_filter_block_by_skip_and_where_clause' has a definition with different parameter names [readability-inconsistent-declaration-parameter-name]
```cpp
Status _filter_block_by_skip_and_where_clause(vectorized::Block* block,
^
```
<details>
<summary>Additional context</summary>
**be/src/vec/sink/vrow_distribution.cpp:114:** the definition seen here
```cpp
Status VRowDistribution::_filter_block_by_skip_and_where_clause(
^
```
**be/src/vec/sink/vrow_distribution.h:112:** differing parameters are named here: ('row_part_tablet_tuple'), in definition: ('row_part_tablet_id')
```cpp
Status _filter_block_by_skip_and_where_clause(vectorized::Block* block,
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798374703
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.03 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162284361 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26432:
URL: https://github.com/apache/doris/pull/26432#issuecomment-1798416380
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.68 seconds
stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162192569 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org