You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2026/05/14 14:09:10 UTC
[PR] [fix](be) Preserve borrowed block data for http send [doris]
mrhhsg opened a new pull request, #63258:
URL: https://github.com/apache/doris/pull/63258
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Broadcast exchange sends can borrow a shared `PBlock` through `set_allocated_block`. When such a request takes the HTTP attachment path, the helper must remove `column_values` from the protobuf request before serialization, but it must not permanently consume data owned by the shared broadcast block. This PR adds an explicit restore mode for borrowed block storage while keeping the default owned-request path unchanged.
It also documents intentional protobuf `set_allocated_*` ownership patterns in exchange sink buffer and rowset meta code, and adds focused unit coverage for the attachment helper's owned vs borrowed behavior.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `./run-be-ut.sh --run --filter=ProtoUtilTest.*`
- Behavior changed: Yes (broadcast exchange HTTP sends preserve borrowed `column_values` for later channel sends)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4619797604
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6208427086
<details>
<summary>ClickBench: <b>Total hot run time: 24.46 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1483f43b2bc9721b2a636a9d7805490469bda5d8, data reload: false
query1 0.01 0.00 0.00
query2 0.09 0.05 0.05
query3 0.26 0.12 0.12
query4 1.60 0.13 0.12
query5 0.27 0.24 0.25
query6 1.18 0.69 0.67
query7 0.03 0.01 0.00
query8 0.06 0.03 0.04
query9 0.38 0.37 0.35
query10 0.60 0.58 0.58
query11 0.20 0.14 0.15
query12 0.18 0.14 0.15
query13 0.48 0.51 0.50
query14 1.03 0.99 1.01
query15 0.62 0.61 0.61
query16 0.35 0.34 0.32
query17 1.11 1.16 1.15
query18 0.22 0.20 0.20
query19 2.12 2.01 1.96
query20 0.02 0.04 0.01
query21 15.43 0.25 0.13
query22 4.54 0.05 0.06
query23 16.29 0.31 0.12
query24 3.03 0.78 0.44
query25 0.13 0.07 0.03
query26 1.76 0.26 0.16
query27 0.04 0.03 0.04
query28 2.07 0.61 0.37
query29 12.54 4.17 3.31
query30 0.28 0.15 0.13
query31 2.78 0.57 0.33
query32 3.22 0.62 0.52
query33 3.15 3.31 3.20
query34 15.41 4.01 3.36
query35 3.36 3.34 3.37
query36 0.55 0.45 0.41
query37 0.10 0.06 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.19 0.14 0.14
query41 0.09 0.03 0.03
query42 0.03 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 95.93 s
Total hot run time: 24.46 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6123080004
Amended to 4494c5ef516 (replaces c8b95c634a7, same scope). A micro-benchmark showed that copying `column_metas` per block on the unicast path cost O(columns) per block (about 29 µs/block at 100 columns in an ASAN build, against 1 µs for the old `Swap`), which was a real regression for wide tables on the default shuffle path. `borrow_block()` therefore now takes a `BlockSource`: OWNED (unicast) blocks hand their metadata over with an O(1) `Swap`, SHARED (broadcast) blocks keep the copy the broadcast multi-block path already did. The payload is never copied in either case, and the http path is unchanged. Also widened `mem_byte` in `_send_rpc` to `int64_t`, since a >= 2 GiB block (the http case that multi-block mode can now send) wrapped the `int`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63258:
URL: https://github.com/apache/doris/pull/63258#discussion_r3242103021
##########
be/src/util/proto_util.h:
##########
@@ -39,9 +40,24 @@ constexpr size_t MIN_HTTP_BRPC_SIZE = (1ULL << 31);
// Embed column_values and brpc request serialization string in controller attachment.
template <typename Params, typename Closure>
Status request_embed_attachment_contain_blockv2(Params* brpc_request,
- std::unique_ptr<Closure>& closure) {
- std::string column_values = std::move(*brpc_request->mutable_block()->mutable_column_values());
- brpc_request->mutable_block()->mutable_column_values()->clear();
+ std::unique_ptr<Closure>& closure,
+ bool restore_column_values = false) {
+ auto* block = brpc_request->mutable_block();
+ if (restore_column_values) {
+ // Some callers borrow block storage from a shared owner. Temporarily detach the large
+ // column_values field so the serialized request stays small, then restore it before
+ // returning so the real owner can still be reused by later sends.
+ auto* column_values = block->release_column_values();
+ DORIS_CHECK(column_values != nullptr);
Review Comment:
This restore mode mutates the borrowed `PBlock` by calling `release_column_values()`, but broadcast blocks are shared across remote channels and can be processed concurrently by different `RpcInstance` callback threads. A concrete path is: an unpartitioned exchange enqueues the same `BroadcastPBlockHolder` to multiple remote channels while those destinations already have in-flight RPCs; when those prior RPCs complete around the same time, each callback calls `_send_rpc()` for its destination and reaches this HTTP helper for the same holder. The first thread temporarily removes `column_values`; the second thread races on the same protobuf and can see `nullptr` here (hitting `DORIS_CHECK`) or otherwise race with the restore. Please avoid mutating the shared `PBlock` during attachment embedding, or add synchronization that covers all users of the shared holder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4452769887
# BE UT Coverage Report
Increment line coverage `5.00% (3/60)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/a5ae6300f8122b1ec603a2727913f39eac342273_a5ae6300f8122b1ec603a2727913f39eac342273/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/a5ae6300f8122b1ec603a2727913f39eac342273_a5ae6300f8122b1ec603a2727913f39eac342273/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.50% (20647/38590) |
| Line Coverage | 37.17% (195210/525119) |
| Region Coverage | 33.54% (152737/455454) |
| Branch Coverage | 34.57% (66595/192650) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4451810981
<details>
<summary>TPC-H: <b>Total hot run time: 29606 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cdde6d13200718593069301b60edc5624d1ea419, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17627 3790 3788 3788
q2 q3 10715 872 598 598
q4 4665 457 341 341
q5 7458 1330 1136 1136
q6 211 169 136 136
q7 915 933 752 752
q8 9716 1387 1308 1308
q9 6078 5372 5345 5345
q10 6308 2064 1814 1814
q11 478 265 256 256
q12 686 424 285 285
q13 18170 3322 2713 2713
q14 296 285 263 263
q15 q16 892 872 796 796
q17 1049 979 786 786
q18 6449 5642 5693 5642
q19 1651 1202 1083 1083
q20 492 399 267 267
q21 4742 2353 1968 1968
q22 468 383 329 329
Total cold run time: 99066 ms
Total hot run time: 29606 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4673 4546 4516 4516
q2 q3 4654 4777 4260 4260
q4 2085 2165 1410 1410
q5 5009 4988 5233 4988
q6 201 172 140 140
q7 2074 1786 1702 1702
q8 3383 3103 3129 3103
q9 8431 8550 8368 8368
q10 4464 4485 4233 4233
q11 629 410 406 406
q12 724 740 528 528
q13 3194 3565 2907 2907
q14 310 312 275 275
q15 q16 763 770 707 707
q17 1347 1318 1279 1279
q18 8091 7026 7046 7026
q19 1167 1159 1156 1156
q20 2255 2242 1987 1987
q21 6115 5435 4748 4748
q22 528 481 395 395
Total cold run time: 60097 ms
Total hot run time: 54134 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6074136835
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63258:
URL: https://github.com/apache/doris/pull/63258#discussion_r4228568846
##########
regression-test/suites/correctness_p0/test_exchange_http_send_borrowed_block.groovy:
##########
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.ClusterOptions
+
+// Forces exchange sinks to use the http-attachment path (normally reserved for
+// requests >= 2G) via a debug point, so that both the unicast (shuffle) and
+// broadcast exchange sink code in proto_util.h/exchange_sink_buffer.cpp that
+// move/borrow column_values into the brpc attachment actually run cross-BE,
+// instead of only through unit tests.
+suite('test_exchange_http_send_borrowed_block', 'docker') {
+ def options = new ClusterOptions()
+ options.beNum = 2
+ options.enableDebugPoints()
+
+ docker(options) {
+ def tbl = 'test_exchange_http_send_borrowed_block_tbl'
+
+ sql "DROP TABLE IF EXISTS ${tbl}"
+ sql """
+ CREATE TABLE ${tbl} (
+ k1 INT NULL,
+ k2 INT NULL
+ )
+ DISTRIBUTED BY HASH(k1) BUCKETS 4
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO ${tbl} VALUES
+ (1, 10), (2, 20), (3, 30), (4, 40), (5, 50), (6, 60), (7, 70), (8, 80)
+ """
+
+ def backends = cluster.getAllBackends()
+ assertTrue(backends.size() == 2)
+ for (def be : backends) {
+ be.enableDebugPoint('proto_util.enable_http_send_block.always_http', null)
+ }
+
+ try {
+ // Shuffle exchange: exercises the unicast attachment path
Review Comment:
[P2] Disable multi-block exchange before these HTTP regression queries. The FE defaults `exchange_multi_blocks_byte_size` to 256 KiB, so `ExchangeSinkBuffer` fills repeated `blocks` instead of the singular `block`. `enable_http_send_block` returns false on `!request.has_block()` before it checks this debug point; both queries therefore use ordinary RPC even when they pass. Set this session variable to a negative value and assert that each forced HTTP branch was reached.
##########
be/src/exec/operator/exchange_sink_buffer.cpp:
##########
@@ -49,6 +49,49 @@
namespace doris {
+namespace exchange_sink_buffer::detail {
+
+void copy_block_metadata_without_column_values(const PBlock& src, PBlock* dst) {
+ for (int i = 0; i < src.column_metas_size(); ++i) {
+ dst->add_column_metas()->CopyFrom(src.column_metas(i));
+ }
+ dst->set_be_exec_version(src.be_exec_version());
+ dst->set_compressed(src.compressed());
+ dst->set_compression_type(src.compression_type());
+ dst->set_uncompressed_size(src.uncompressed_size());
+}
+
+std::shared_ptr<PTransmitDataParams> make_http_request_without_column_values(
+ const PTransmitDataParams& src) {
+ DORIS_CHECK(src.has_block());
+ DORIS_CHECK(src.blocks_size() == 0);
+ DORIS_CHECK(!src.has_row_batch());
+
+ auto dst = std::make_shared<PTransmitDataParams>();
+ dst->mutable_finst_id()->CopyFrom(src.finst_id());
+ dst->set_node_id(src.node_id());
+ dst->set_sender_id(src.sender_id());
+ dst->set_be_number(src.be_number());
+ dst->set_eos(src.eos());
+ dst->set_packet_seq(src.packet_seq());
+ if (src.has_query_statistics()) {
+ dst->mutable_query_statistics()->CopyFrom(src.query_statistics());
+ }
+ if (src.has_transfer_by_attachment()) {
+ dst->set_transfer_by_attachment(src.transfer_by_attachment());
+ }
+ if (src.has_query_id()) {
+ dst->mutable_query_id()->CopyFrom(src.query_id());
+ }
+ if (src.has_exec_status()) {
+ dst->mutable_exec_status()->CopyFrom(src.exec_status());
+ }
+ copy_block_metadata_without_column_values(src.block(), dst->mutable_block());
Review Comment:
[P1] Preserve `column_values` presence in the HTTP metadata request. This new `PBlock` omits the optional field entirely. On receive, `attachment_extract_request_contain_block` takes a mutable string pointer before `ParseFromString`, then writes the attachment through that pointer; the parse clears the proto2 presence bit, and the raw string write does not restore it. `VDataStreamMgr::transmit_block` then `CopyFrom`s the block without its payload, so large broadcast sends fail or deserialize empty data. Mark the field present with an empty value in this request, or set it through the protobuf API after extraction, and cover the receive-plus-`CopyFrom` path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6137155260
<details>
<summary>TPC-H: <b>Total hot run time: 28786 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a85aea5c7cf32ebf093fd8b3069a828cb0710e5, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17571 5662 5568 5568
q2 2096 326 271 271
q3 10216 1391 838 838
q4 4675 475 352 352
q5 7416 808 535 535
q6 186 189 154 154
q7 782 820 615 615
q8 9244 1337 1059 1059
q9 5687 4455 4421 4421
q10 6831 1303 1020 1020
q11 426 248 230 230
q12 623 404 289 289
q13 18055 3072 2413 2413
q14 281 271 255 255
q15 q16 732 725 671 671
q17 1184 745 563 563
q18 7157 6233 6203 6203
q19 1100 961 591 591
q20 375 330 222 222
q21 4415 2250 2395 2250
q22 360 312 266 266
Total cold run time: 99412 ms
Total hot run time: 28786 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6148 6148 6108 6108
q2 699 556 561 556
q3 5276 5435 4933 4933
q4 2006 2128 1582 1582
q5 5291 5189 6033 5189
q6 252 193 154 154
q7 2304 2131 1861 1861
q8 3168 2793 2667 2667
q9 8346 8140 8446 8140
q10 3799 3722 3347 3347
q11 655 441 411 411
q12 755 746 516 516
q13 2979 3213 2536 2536
q14 321 319 291 291
q15 q16 706 705 672 672
q17 8429 7541 7351 7351
q18 13177 12279 13130 12279
q19 921 796 788 788
q20 2200 2180 1941 1941
q21 5799 4729 4799 4729
q22 504 485 409 409
Total cold run time: 73735 ms
Total hot run time: 66460 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6209491580
# BE UT Coverage Report
Increment line coverage `83.67% (82/98)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1483f43b2bc9721b2a636a9d7805490469bda5d8_1483f43b2bc9721b2a636a9d7805490469bda5d8/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1483f43b2bc9721b2a636a9d7805490469bda5d8_1483f43b2bc9721b2a636a9d7805490469bda5d8/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 66.35% (31690/47763) |
| Line Coverage | 51.10% (333653/652917) |
| Region Coverage | 46.44% (269552/580394) |
| Branch Coverage | 48.12% (126817/263520) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4452295873
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6083337599
# BE Regression && UT Coverage Report
Increment line coverage `7.14% (1/14)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63258_8d44f49991e4bea859b3bdcb1d2f342c3a507aea_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63258_8d44f49991e4bea859b3bdcb1d2f342c3a507aea_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.72% (35225/45913) |
| Line Coverage | 61.94% (398310/643053) |
| Region Coverage | 58.40% (336434/576110) |
| Branch Coverage | 59.25% (154669/261034) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6081514032
<details>
<summary>TPC-H: <b>Total hot run time: 29646 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8d44f49991e4bea859b3bdcb1d2f342c3a507aea, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17610 5786 5665 5665
q2 2093 293 250 250
q3 10263 1425 865 865
q4 4677 486 354 354
q5 7443 822 539 539
q6 207 201 165 165
q7 796 816 615 615
q8 9268 1384 1205 1205
q9 5760 4634 4530 4530
q10 6829 1286 1010 1010
q11 441 261 242 242
q12 636 406 296 296
q13 18032 3252 2479 2479
q14 281 279 252 252
q15 q16 752 745 683 683
q17 1181 733 589 589
q18 7138 6220 6177 6177
q19 1102 964 603 603
q20 411 349 232 232
q21 5384 2586 2658 2586
q22 412 410 309 309
Total cold run time: 100716 ms
Total hot run time: 29646 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6709 6653 6488 6488
q2 834 644 593 593
q3 5529 5731 5191 5191
q4 2252 2553 1756 1756
q5 5842 5665 5619 5619
q6 253 205 159 159
q7 2396 2131 1892 1892
q8 3224 2804 2877 2804
q9 8130 7799 7693 7693
q10 3799 3686 3314 3314
q11 628 435 407 407
q12 742 755 565 565
q13 2936 3202 2471 2471
q14 288 285 262 262
q15 q16 707 722 650 650
q17 7952 7189 6904 6904
q18 13242 12284 13107 12284
q19 965 833 841 833
q20 2226 2204 1980 1980
q21 6150 4978 5157 4978
q22 529 558 428 428
Total cold run time: 75333 ms
Total hot run time: 67271 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6196692252
/review-light
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6190897357
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6191904883
<details>
<summary>TPC-DS: <b>Total hot run time: 152417 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e8fd6bc4b1a1badf954c57536ee02456b4d4da8, data reload: false
query5 4307 601 471 471
query6 435 219 209 209
query7 4835 451 230 230
query8 324 166 156 156
query9 8778 3967 3938 3938
query10 457 321 251 251
query11 6039 3818 3531 3531
query12 153 90 89 89
query13 1242 434 322 322
query14 6116 4005 3710 3710
query14_1 3491 3468 3520 3468
query15 201 194 181 181
query16 972 424 377 377
query17 838 667 538 538
query18 2426 435 316 316
query19 207 162 138 138
query20 86 93 85 85
query21 210 141 114 114
query22 13028 13092 12894 12894
query23 13385 12806 12428 12428
query23_1 12719 12482 12560 12482
query24 7144 829 456 456
query24_1 470 478 472 472
query25 513 408 337 337
query26 1094 256 134 134
query27 2749 432 264 264
query28 4603 1961 1936 1936
query29 1425 578 452 452
query30 303 219 184 184
query31 991 930 796 796
query32 140 93 92 92
query33 516 298 238 238
query34 879 840 492 492
query35 745 753 667 667
query36 829 797 721 721
query37 132 96 91 91
query38 1787 1746 1691 1691
query39 798 825 759 759
query39_1 741 729 742 729
query40 215 111 93 93
query41 71 67 64 64
query42 89 86 83 83
query43 369 373 328 328
query44 1278 705 715 705
query45 183 183 172 172
query46 845 969 568 568
query47 2738 2968 2949 2949
query48 301 307 224 224
query49 590 400 304 304
query50 682 284 221 221
query51 10632 10513 10360 10360
query52 89 79 80 79
query53 188 209 160 160
query54 268 204 211 204
query55 80 72 66 66
query56 263 244 222 222
query57 1634 1594 1555 1555
query58 298 272 269 269
query59 2174 2161 1925 1925
query60 309 255 244 244
query61 172 170 170 170
query62 397 342 298 298
query63 199 164 169 164
query64 2598 1073 898 898
query65 3287 3234 3261 3234
query66 1762 435 336 336
query67 20104 20062 20046 20046
query68 2997 1005 619 619
query69 396 299 263 263
query70 848 794 776 776
query71 301 232 215 215
query72 2669 2587 2239 2239
query73 521 518 294 294
query74 4609 4455 4324 4324
query75 2339 2299 1931 1931
query76 2301 1015 623 623
query77 337 372 286 286
query78 9789 9765 9208 9208
query79 968 905 518 518
query80 1086 421 357 357
query81 578 333 274 274
query82 592 149 116 116
query83 281 200 177 177
query84 309 126 101 101
query85 1016 441 393 393
query86 361 241 234 234
query87 2006 1964 1830 1830
query88 3626 2738 2695 2695
query89 339 299 265 265
query90 1714 188 182 182
query91 164 151 132 132
query92 103 80 92 80
query93 1030 1020 593 593
query94 513 312 288 288
query95 619 355 383 355
query96 679 529 232 232
query97 2445 2440 2317 2317
query98 160 152 148 148
query99 785 777 648 648
Total cold run time: 234196 ms
Total hot run time: 152417 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6136863692
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6139667796
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4621506301
<details>
<summary>TPC-H: <b>Total hot run time: 29677 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5671323e79e6951c90a481990d512052023b5ca0, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17688 4198 4202 4198
q2 q3 10744 1441 850 850
q4 4691 485 349 349
q5 7576 898 593 593
q6 188 188 146 146
q7 825 858 637 637
q8 9372 1598 1714 1598
q9 5876 4539 4560 4539
q10 6753 1838 1533 1533
q11 437 278 260 260
q12 629 431 309 309
q13 18136 3428 2705 2705
q14 273 263 249 249
q15 q16 806 783 707 707
q17 987 884 968 884
q18 7068 5805 5604 5604
q19 1311 1378 1119 1119
q20 508 416 265 265
q21 6369 2872 2817 2817
q22 456 382 315 315
Total cold run time: 100693 ms
Total hot run time: 29677 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5142 4956 5038 4956
q2 q3 4967 5296 4777 4777
q4 2147 2202 1402 1402
q5 4940 5047 4801 4801
q6 254 194 138 138
q7 1922 1807 1689 1689
q8 2525 2177 2199 2177
q9 8185 7765 7574 7574
q10 4792 4701 4307 4307
q11 551 387 354 354
q12 751 761 527 527
q13 3002 3468 2803 2803
q14 273 281 256 256
q15 q16 687 694 600 600
q17 1292 1276 1271 1271
q18 7260 6955 6684 6684
q19 1125 1088 1139 1088
q20 2228 2223 1954 1954
q21 5286 4586 4466 4466
q22 527 445 398 398
Total cold run time: 57856 ms
Total hot run time: 52222 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4451420256
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4458604820
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6079052820
/review-light
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6190682399
/review-light
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63258:
URL: https://github.com/apache/doris/pull/63258#discussion_r4286809662
##########
be/src/util/proto_util.h:
##########
@@ -36,26 +42,131 @@ namespace doris {
// 2G: In the default "baidu_std" brpcd, upper limit of the request and attachment length is 2G.
constexpr size_t MIN_HTTP_BRPC_SIZE = (1ULL << 31);
-// Embed column_values and brpc request serialization string in controller attachment.
+// Attachment layout shared by every http send path:
+//
+// [int64 request_size][serialized request][int64 payload_size][payload]...
+//
+// The serialized request carries everything but the block payloads (PBlock::column_values).
+// One payload section follows per payload block of the request, in block order, see
+// payload_blocks(). A request with a single block therefore keeps the layout older
+// receivers expect; receivers older than this layout read exactly one payload into the
+// singular `block`, so a multi-payload attachment (repeated `blocks`) is only understood by
+// receivers that carry it. Such requests could not be sent at all before (plain brpc rejected
+// them for exceeding its size limit), so no working mixed-version case is lost.
+
+// The blocks whose column_values travel as attachment payloads. A generic request carries one
+// block; PTransmitDataParams carries either the singular `block` or the repeated `blocks`.
+template <typename Params>
+std::vector<PBlock*> payload_blocks(Params* brpc_request) {
+ return {brpc_request->mutable_block()};
+}
+
+inline std::vector<PBlock*> payload_blocks(PTransmitDataParams* brpc_request) {
+ if (brpc_request->has_block()) {
+ DCHECK(brpc_request->blocks().empty());
+ return {brpc_request->mutable_block()};
+ }
+ std::vector<PBlock*> blocks;
+ blocks.reserve(brpc_request->blocks_size());
+ for (auto& block : *brpc_request->mutable_blocks()) {
+ blocks.push_back(&block);
+ }
+ return blocks;
+}
+
+// Append one `[int64 size][bytes]` section to the attachment.
+inline Status attachment_append_section(butil::IOBuf* attachment, const std::string& data) {
+ int64_t data_size = data.size();
+ attachment->append(&data_size, sizeof(data_size));
+ try {
+ attachment->append(data);
+ } catch (...) {
+ LOG(WARNING) << "Try to alloc " << data_size
+ << " bytes for append data to attachment failed. ";
+ return Status::MemoryAllocFailed("request embed attachment failed to memcpy {} bytes",
+ data_size);
+ }
+ return Status::OK();
+}
+
+// Embed `brpc_request` followed by `payloads`, one section each, in the controller attachment.
+template <typename Params, typename Closure>
+Status request_embed_attachmentv2(Params* brpc_request,
+ const std::vector<const std::string*>& payloads,
+ std::unique_ptr<Closure>& closure) {
+ std::string req_str;
+ if (!brpc_request->SerializeToString(&req_str)) {
+ return Status::InternalError("failed to serialize the request");
+ }
+ butil::IOBuf attachment;
+ RETURN_IF_ERROR(attachment_append_section(&attachment, req_str));
+ for (const auto* payload : payloads) {
+ RETURN_IF_ERROR(attachment_append_section(&attachment, *payload));
+ }
+ closure->cntl_->request_attachment().swap(attachment);
+ return Status::OK();
+}
+
+template <typename Params, typename Closure>
+Status request_embed_attachmentv2(Params* brpc_request, const std::string& data,
+ std::unique_ptr<Closure>& closure) {
+ return request_embed_attachmentv2(brpc_request, std::vector<const std::string*> {&data},
+ closure);
+}
+
+// Embed a request that owns its block: column_values is moved into the attachment and the
+// request keeps the field present but empty.
template <typename Params, typename Closure>
Status request_embed_attachment_contain_blockv2(Params* brpc_request,
std::unique_ptr<Closure>& closure) {
- std::string column_values = std::move(*brpc_request->mutable_block()->mutable_column_values());
- brpc_request->mutable_block()->mutable_column_values()->clear();
+ auto* block = brpc_request->mutable_block();
+ std::string column_values = std::move(*block->mutable_column_values());
+ block->mutable_column_values()->clear();
return request_embed_attachmentv2(brpc_request, column_values, closure);
}
+// Embed a request whose blocks only borrow their column_values (see
+// ExchangeSinkBuffer / borrow_block()). Each payload is detached from the request while the
+// request is serialized, appended by reference and attached back, so the owner of the payload
+// is never read through the request's presence bits nor mutated: the same owner may be
+// serialized by other channels at the same time.
+template <typename Closure>
+Status request_embed_attachment_borrowed_blocksv2(PTransmitDataParams* brpc_request,
+ std::unique_ptr<Closure>& closure) {
+ std::vector<PBlock*> blocks = payload_blocks(brpc_request);
+ std::vector<std::string*> payloads;
+ payloads.reserve(blocks.size());
+ // Registered before any payload is detached: if a placeholder allocation below throws, the
+ // payloads detached so far are attached back and the placeholders already installed are freed
+ // by set_allocated_column_values().
+ Defer attach_back([&]() {
+ for (size_t i = 0; i < payloads.size(); ++i) {
+ blocks[i]->set_allocated_column_values(payloads[i]);
+ }
+ });
+ for (auto* block : blocks) {
+ payloads.push_back(block->release_column_values());
+ DCHECK(payloads.back() != nullptr);
+ // Keep the field present on the wire: receivers that take the payload slot before
+ // parsing the request rely on the serialized field to restore its presence bit.
+ block->set_column_values("");
+ }
+ return request_embed_attachmentv2(
+ brpc_request, std::vector<const std::string*>(payloads.begin(), payloads.end()),
+ closure);
+}
+
inline bool enable_http_send_block(const PTransmitDataParams& request) {
if (!config::transfer_large_data_by_brpc) {
return false;
}
- if (!request.has_block() || !request.block().has_column_values()) {
- return false;
- }
- if (request.ByteSizeLong() < MIN_HTTP_BRPC_SIZE) {
+ if (!request.has_block() && request.blocks().empty()) {
Review Comment:
[P1] Keep repeated-block HTTP compatible with older receivers
During a rolling upgrade, a new sender with multi-block exchange enabled can send a >=2 GiB request through this branch to a BE still running the base version. That receiver's `attachment_extract_request_contain_block()` only fills the singular `block.column_values`; it parses the repeated `blocks` but leaves their payloads empty. `VDataStreamMgr::transmit_block()` then selects those repeated blocks and enqueues them. With the default `fragment_transmission_compression_codec = 'none'`, `Block::deserialize()` passes these empty buffers to unchecked column header decoding (`deserialize_const_flag_and_row_num`), causing invalid reads/decoding instead of the former transport rejection. Preserve the legacy singular framing, negotiate receiver support, or fail before emitting the new shape. This differs from the fixed singular presence-bit thread: setting the empty repeated fields present does not teach the old receiver to attach their data.
##########
be/src/exec/operator/exchange_sink_buffer.cpp:
##########
@@ -477,35 +490,14 @@ Status ExchangeSinkBuffer::_send_rpc(RpcInstance& instance_data) {
return !req.block_holder->serialized_block().empty() ||
req.block_holder->get_block()->column_metas().empty();
});
- auto& request = requests[0];
auto& brpc_request = instance_data.request;
brpc_request->set_sender_id(channel->_parent->sender_id());
brpc_request->set_be_number(channel->_parent->be_number());
- if (_send_multi_blocks && !use_shared_protobuf) {
- for (const auto& req : requests) {
- if (auto block = req.block_holder->get_block();
- block && !block->column_metas().empty()) {
- add_borrowed_broadcast_block(*brpc_request, *block);
- }
- }
- } else if (!_send_multi_blocks) {
- if (request.block_holder->get_block() &&
- !request.block_holder->get_block()->column_metas().empty()) {
- brpc_request->set_allocated_block(request.block_holder->get_block());
- }
- }
- auto detach_blocks = [&]() {
- if (!_send_multi_blocks && request.block_holder->get_block()) {
- static_cast<void>(brpc_request->release_block());
- } else {
- for (int i = 0; i < brpc_request->mutable_blocks()->size(); ++i) {
- static_cast<void>(brpc_request->mutable_blocks(i)->release_column_values());
- }
- brpc_request->clear_blocks();
- }
- };
- Defer release_block(detach_blocks);
+ Defer release_blocks([&]() { release_borrowed_blocks(brpc_request.get()); });
+ borrow_blocks(
Review Comment:
[P2] Skip rebuilding blocks for cached broadcast sends
When `_send_multi_blocks` and `use_shared_protobuf` are true, this new unconditional call deep-copies every block's recursive `column_metas` into a request for each destination. For normal requests below 2 GiB, the cached branch immediately clears these blocks and sends the pre-serialized holder fragments instead. The previous branch skipped construction in this case. A wide/nested-schema broadcast with many remote destinations and small batches therefore adds a full schema-copy pass per destination under the RPC mutex for metadata that is never sent. Compute the HTTP decision from the routing header and cached fragment sizes, and construct borrowed blocks only for HTTP or the uncached fallback. This is separate from the earlier metadata-ownership simplification: no request uses these copies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6191992232
<details>
<summary>ClickBench: <b>Total hot run time: 24.8 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e8fd6bc4b1a1badf954c57536ee02456b4d4da8, data reload: false
query1 0.00 0.00 0.01
query2 0.10 0.04 0.05
query3 0.27 0.12 0.12
query4 1.62 0.12 0.13
query5 0.27 0.25 0.25
query6 1.17 0.70 0.68
query7 0.04 0.01 0.00
query8 0.06 0.04 0.04
query9 0.41 0.32 0.34
query10 0.57 0.62 0.59
query11 0.20 0.14 0.14
query12 0.19 0.15 0.15
query13 0.49 0.50 0.50
query14 1.03 1.01 1.01
query15 0.64 0.63 0.62
query16 0.31 0.31 0.34
query17 1.10 1.14 1.14
query18 0.23 0.22 0.20
query19 2.11 1.99 2.07
query20 0.02 0.02 0.01
query21 15.42 0.29 0.14
query22 4.34 0.06 0.06
query23 16.27 0.31 0.12
query24 3.24 0.64 0.45
query25 0.11 0.06 0.05
query26 1.41 0.27 0.16
query27 0.05 0.03 0.03
query28 1.87 0.61 0.38
query29 12.50 4.25 3.35
query30 0.29 0.14 0.13
query31 2.78 0.58 0.34
query32 3.23 0.61 0.52
query33 3.17 3.26 3.30
query34 15.49 4.06 3.44
query35 3.35 3.32 3.37
query36 0.54 0.46 0.46
query37 0.10 0.07 0.07
query38 0.04 0.03 0.04
query39 0.04 0.03 0.02
query40 0.18 0.14 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.04 0.04
Total cold run time: 95.42 s
Total hot run time: 24.8 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4452213818
# BE UT Coverage Report
Increment line coverage `73.33% (11/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/cdde6d13200718593069301b60edc5624d1ea419_cdde6d13200718593069301b60edc5624d1ea419/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/cdde6d13200718593069301b60edc5624d1ea419_cdde6d13200718593069301b60edc5624d1ea419/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.50% (20645/38587) |
| Line Coverage | 37.16% (195113/525081) |
| Region Coverage | 33.55% (152792/455422) |
| Branch Coverage | 34.56% (66576/192633) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4453606635
# BE Regression && UT Coverage Report
Increment line coverage `6.67% (4/60)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63258_a5ae6300f8122b1ec603a2727913f39eac342273_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63258_a5ae6300f8122b1ec603a2727913f39eac342273_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.62% (27820/37790) |
| Line Coverage | 57.53% (301306/523742) |
| Region Coverage | 54.66% (251346/459869) |
| Branch Coverage | 56.24% (108763/193380) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4458717576
<details>
<summary>TPC-H: <b>Total hot run time: 31185 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5671323e79e6951c90a481990d512052023b5ca0, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17633 3876 3849 3849
q2 q3 10808 1384 799 799
q4 4690 466 348 348
q5 7628 2326 2112 2112
q6 239 176 137 137
q7 911 780 662 662
q8 9358 1908 1581 1581
q9 5108 4912 4886 4886
q10 6453 2058 1823 1823
q11 431 270 244 244
q12 633 422 293 293
q13 18111 3416 2784 2784
q14 265 250 237 237
q15 q16 818 767 708 708
q17 831 925 907 907
q18 6915 5728 5586 5586
q19 1316 1192 1060 1060
q20 505 392 259 259
q21 6511 2838 2590 2590
q22 462 376 320 320
Total cold run time: 99626 ms
Total hot run time: 31185 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4548 4547 4422 4422
q2 q3 4853 5162 4569 4569
q4 2111 2187 1398 1398
q5 4915 4536 4693 4536
q6 250 190 135 135
q7 1900 1700 1544 1544
q8 2353 2015 2196 2015
q9 7732 7245 7201 7201
q10 4486 4402 4003 4003
q11 525 381 351 351
q12 708 730 509 509
q13 2966 3367 2789 2789
q14 279 271 249 249
q15 q16 671 694 619 619
q17 1258 1233 1228 1228
q18 7219 6664 6773 6664
q19 1115 1065 1076 1065
q20 2208 2210 1925 1925
q21 5295 4581 4431 4431
q22 515 451 424 424
Total cold run time: 55907 ms
Total hot run time: 50077 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4452426368
<details>
<summary>TPC-H: <b>Total hot run time: 29615 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a5ae6300f8122b1ec603a2727913f39eac342273, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17607 3800 3748 3748
q2 q3 10700 865 611 611
q4 4659 465 348 348
q5 7445 1320 1132 1132
q6 185 163 134 134
q7 924 950 754 754
q8 9301 1427 1313 1313
q9 5587 5356 5318 5318
q10 6247 2086 1819 1819
q11 459 258 263 258
q12 627 403 295 295
q13 18080 3212 2753 2753
q14 290 282 261 261
q15 q16 897 857 786 786
q17 1020 1002 764 764
q18 6466 5688 5677 5677
q19 1186 1233 1027 1027
q20 511 407 251 251
q21 5357 2399 2022 2022
q22 488 396 344 344
Total cold run time: 98036 ms
Total hot run time: 29615 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4696 4697 4808 4697
q2 q3 4620 4818 4179 4179
q4 2106 2151 1400 1400
q5 4939 4961 5222 4961
q6 203 163 133 133
q7 2107 1780 1598 1598
q8 3309 3062 3111 3062
q9 8474 8397 8458 8397
q10 4497 4539 4262 4262
q11 581 426 383 383
q12 688 746 524 524
q13 3237 3583 2903 2903
q14 314 313 286 286
q15 q16 760 760 674 674
q17 1475 1310 1241 1241
q18 7923 7106 6935 6935
q19 1127 1160 1145 1145
q20 2217 2233 1940 1940
q21 6118 5325 4837 4837
q22 546 512 421 421
Total cold run time: 59937 ms
Total hot run time: 53978 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6140347343
<details>
<summary>TPC-DS: <b>Total hot run time: 151796 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8061f1e2c3f5727e29678cf9157fa670a52e4f6, data reload: false
query5 4319 582 447 447
query6 424 211 197 197
query7 4816 437 223 223
query8 317 182 176 176
query9 8778 3955 3949 3949
query10 456 301 250 250
query11 6003 3411 3176 3176
query12 140 91 81 81
query13 1251 453 344 344
query14 6497 4761 4385 4385
query14_1 4197 4252 4236 4236
query15 208 206 177 177
query16 944 472 438 438
query17 851 651 543 543
query18 2412 420 322 322
query19 200 164 162 162
query20 78 75 78 75
query21 208 131 107 107
query22 13041 13008 12837 12837
query23 13027 12519 12152 12152
query23_1 12215 12318 12092 12092
query24 7083 814 456 456
query24_1 464 460 465 460
query25 506 392 337 337
query26 1316 257 138 138
query27 2746 430 266 266
query28 4643 1934 1937 1934
query29 1559 568 428 428
query30 298 218 181 181
query31 893 771 635 635
query32 135 93 88 88
query33 520 294 231 231
query34 961 856 490 490
query35 736 768 653 653
query36 839 782 786 782
query37 130 100 92 92
query38 1781 1743 1702 1702
query39 720 721 653 653
query39_1 665 662 685 662
query40 217 124 97 97
query41 68 64 66 64
query42 82 79 90 79
query43 359 375 322 322
query44 1301 707 692 692
query45 181 176 159 159
query46 850 947 561 561
query47 2929 3031 2728 2728
query48 305 335 224 224
query49 570 423 294 294
query50 678 284 213 213
query51 10551 10592 10674 10592
query52 78 85 67 67
query53 193 210 150 150
query54 247 181 180 180
query55 75 67 62 62
query56 222 207 211 207
query57 1626 1568 1558 1558
query58 276 258 251 251
query59 2359 2411 2116 2116
query60 286 235 219 219
query61 151 150 146 146
query62 394 339 286 286
query63 191 156 162 156
query64 2747 957 768 768
query65 3418 3346 3353 3346
query66 1778 416 303 303
query67 20194 20057 20101 20057
query68 2909 1011 614 614
query69 400 293 254 254
query70 906 827 816 816
query71 287 234 215 215
query72 2894 2735 2388 2388
query73 514 527 298 298
query74 4601 4482 4297 4297
query75 2339 2302 1942 1942
query76 2315 1015 607 607
query77 352 410 304 304
query78 8993 8963 8502 8502
query79 972 896 521 521
query80 611 414 353 353
query81 535 317 270 270
query82 571 134 107 107
query83 303 189 176 176
query84 308 120 97 97
query85 816 415 363 363
query86 326 248 241 241
query87 2007 1949 1815 1815
query88 3587 2697 2648 2648
query89 332 292 256 256
query90 1902 182 180 180
query91 159 150 122 122
query92 97 78 88 78
query93 987 993 561 561
query94 457 301 275 275
query95 584 406 332 332
query96 666 515 232 232
query97 2473 2429 2340 2340
query98 163 149 142 142
query99 782 769 660 660
Total cold run time: 233589 ms
Total hot run time: 151796 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6140205805
<details>
<summary>TPC-H: <b>Total hot run time: 28903 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8061f1e2c3f5727e29678cf9157fa670a52e4f6, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17618 5680 5525 5525
q2 2112 311 257 257
q3 10241 1394 827 827
q4 4677 481 344 344
q5 7423 791 538 538
q6 185 192 158 158
q7 776 796 609 609
q8 9254 1290 1086 1086
q9 5658 4418 4433 4418
q10 6822 1298 1017 1017
q11 434 253 236 236
q12 634 399 283 283
q13 18065 3062 2378 2378
q14 271 270 249 249
q15 q16 734 718 675 675
q17 1154 733 571 571
q18 7056 6229 6165 6165
q19 1113 950 582 582
q20 379 335 227 227
q21 5523 2490 2552 2490
q22 399 342 268 268
Total cold run time: 100528 ms
Total hot run time: 28903 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6427 6495 6449 6449
q2 738 574 562 562
q3 5583 5744 5212 5212
q4 2041 2444 1574 1574
q5 5828 5658 5594 5594
q6 252 186 148 148
q7 2279 2050 1808 1808
q8 3101 2700 2630 2630
q9 8310 7992 7816 7816
q10 3678 3584 3192 3192
q11 595 403 399 399
q12 666 693 491 491
q13 2730 3054 2391 2391
q14 290 287 255 255
q15 q16 678 672 615 615
q17 7829 7100 6879 6879
q18 13149 12226 13034 12226
q19 901 784 774 774
q20 2179 2137 1942 1942
q21 5746 4656 4741 4656
q22 493 452 402 402
Total cold run time: 73493 ms
Total hot run time: 66015 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6208263174
<details>
<summary>TPC-H: <b>Total hot run time: 29200 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1483f43b2bc9721b2a636a9d7805490469bda5d8, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17574 5656 5512 5512
q2 2155 311 262 262
q3 10197 1402 845 845
q4 4685 473 349 349
q5 7429 819 526 526
q6 188 188 160 160
q7 765 806 617 617
q8 9269 1383 1099 1099
q9 5501 4277 4264 4264
q10 6819 1296 1028 1028
q11 453 256 231 231
q12 633 402 298 298
q13 18094 3567 3037 3037
q14 280 266 249 249
q15 q16 738 725 650 650
q17 1203 752 573 573
q18 7036 6205 6183 6183
q19 1111 951 583 583
q20 383 332 220 220
q21 5091 2252 2538 2252
q22 365 309 262 262
Total cold run time: 99969 ms
Total hot run time: 29200 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6119 6120 6115 6115
q2 687 597 538 538
q3 5241 5397 4959 4959
q4 1993 2108 1543 1543
q5 5287 5254 6151 5254
q6 263 200 156 156
q7 2327 2096 1921 1921
q8 3323 2929 2782 2782
q9 8220 8162 8091 8091
q10 3750 3759 3340 3340
q11 638 423 393 393
q12 691 750 509 509
q13 3463 3905 3310 3310
q14 338 315 284 284
q15 q16 716 712 659 659
q17 8326 7568 7414 7414
q18 13192 12279 13119 12279
q19 921 789 794 789
q20 2184 2163 1940 1940
q21 5782 4726 4786 4726
q22 497 446 403 403
Total cold run time: 73958 ms
Total hot run time: 67405 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6077855065
/review-light
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4620919658
# BE Regression && UT Coverage Report
Increment line coverage `62.90% (39/62)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63258_5671323e79e6951c90a481990d512052023b5ca0_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63258_5671323e79e6951c90a481990d512052023b5ca0_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.80% (27469/38258) |
| Line Coverage | 55.37% (294110/531214) |
| Region Coverage | 52.13% (245457/470831) |
| Branch Coverage | 53.25% (106061/199168) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6189823526
/review-light
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6137263563
<details>
<summary>TPC-DS: <b>Total hot run time: 153228 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a85aea5c7cf32ebf093fd8b3069a828cb0710e5, data reload: false
query5 4330 630 458 458
query6 434 200 191 191
query7 4797 454 228 228
query8 318 183 165 165
query9 8787 4059 4052 4052
query10 466 312 257 257
query11 5986 3440 3198 3198
query12 142 86 84 84
query13 1241 460 326 326
query14 6543 4794 4524 4524
query14_1 4321 4292 4291 4291
query15 216 208 184 184
query16 919 442 422 422
query17 866 667 558 558
query18 2428 427 317 317
query19 193 162 138 138
query20 87 82 78 78
query21 208 139 112 112
query22 13158 13096 13034 13034
query23 13207 12611 12229 12229
query23_1 12374 12370 12331 12331
query24 7100 860 475 475
query24_1 496 487 480 480
query25 543 414 377 377
query26 1172 256 146 146
query27 2737 448 276 276
query28 4563 1989 1984 1984
query29 1494 621 471 471
query30 298 219 183 183
query31 877 775 656 656
query32 141 93 95 93
query33 524 342 245 245
query34 902 857 508 508
query35 768 793 654 654
query36 834 797 717 717
query37 135 98 92 92
query38 1783 1742 1694 1694
query39 702 712 712 712
query39_1 693 661 682 661
query40 214 114 98 98
query41 69 63 62 62
query42 85 85 84 84
query43 366 381 327 327
query44 1377 700 703 700
query45 176 180 163 163
query46 866 966 565 565
query47 2941 2955 2988 2955
query48 299 330 226 226
query49 575 392 292 292
query50 718 285 219 219
query51 10604 10422 10292 10292
query52 81 84 73 73
query53 198 224 160 160
query54 243 195 205 195
query55 73 69 67 67
query56 241 210 209 209
query57 1613 1604 1572 1572
query58 286 261 250 250
query59 2367 2370 2131 2131
query60 288 237 225 225
query61 147 145 138 138
query62 391 347 293 293
query63 205 178 174 174
query64 2488 934 768 768
query65 3455 3358 3381 3358
query66 1781 438 305 305
query67 20335 20290 20068 20068
query68 2940 1033 618 618
query69 375 274 252 252
query70 943 856 819 819
query71 291 238 219 219
query72 2679 2581 2492 2492
query73 522 539 296 296
query74 4593 4501 4336 4336
query75 2344 2310 1964 1964
query76 2314 1031 629 629
query77 377 397 297 297
query78 9002 9009 8537 8537
query79 1040 892 542 542
query80 526 435 335 335
query81 520 321 271 271
query82 577 139 112 112
query83 300 201 188 188
query84 312 128 104 104
query85 798 463 357 357
query86 312 239 234 234
query87 2021 1982 1843 1843
query88 3670 2711 2695 2695
query89 339 295 257 257
query90 1889 192 183 183
query91 159 153 131 131
query92 89 86 87 86
query93 1051 1019 583 583
query94 458 309 275 275
query95 585 408 326 326
query96 675 539 235 235
query97 2473 2426 2301 2301
query98 155 148 143 143
query99 783 767 667 667
Total cold run time: 234184 ms
Total hot run time: 153228 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6137316436
<details>
<summary>ClickBench: <b>Total hot run time: 24.63 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a85aea5c7cf32ebf093fd8b3069a828cb0710e5, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.04 0.05
query3 0.26 0.12 0.12
query4 1.60 0.12 0.12
query5 0.28 0.25 0.25
query6 1.17 0.68 0.69
query7 0.04 0.01 0.00
query8 0.06 0.04 0.04
query9 0.42 0.35 0.35
query10 0.58 0.60 0.57
query11 0.20 0.14 0.13
query12 0.19 0.14 0.14
query13 0.49 0.50 0.52
query14 0.98 0.96 0.96
query15 0.65 0.61 0.61
query16 0.36 0.34 0.34
query17 1.16 1.10 1.11
query18 0.24 0.20 0.20
query19 2.12 1.97 1.98
query20 0.02 0.01 0.01
query21 15.42 0.28 0.14
query22 4.46 0.05 0.05
query23 16.28 0.33 0.13
query24 3.18 0.59 0.42
query25 0.12 0.05 0.04
query26 1.32 0.25 0.16
query27 0.04 0.03 0.05
query28 1.69 0.62 0.42
query29 12.55 4.15 3.31
query30 0.29 0.14 0.13
query31 2.76 0.58 0.33
query32 3.22 0.62 0.53
query33 3.15 3.27 3.25
query34 15.46 4.00 3.38
query35 3.34 3.36 3.37
query36 0.54 0.48 0.46
query37 0.09 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.16 0.14 0.14
query41 0.08 0.04 0.03
query42 0.04 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 95.25 s
Total hot run time: 24.63 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63258:
URL: https://github.com/apache/doris/pull/63258#discussion_r4254638430
##########
be/src/exec/operator/exchange_sink_buffer.cpp:
##########
@@ -49,6 +49,52 @@
namespace doris {
+void borrow_block(PBlock* src, PBlock* dst, BlockSource source) {
+ if (source == BlockSource::OWNED) {
+ // O(1) pointer swap; the owner is destroyed right after the send and takes the request's
+ // cleared metas with it.
+ dst->mutable_column_metas()->Swap(src->mutable_column_metas());
+ } else {
+ dst->mutable_column_metas()->CopyFrom(src->column_metas());
+ }
+ // set_allocated_column_values() stores the owner's std::string pointer as is; the thirdparty
+ // protobuf is built without NDEBUG checks, otherwise ArenaStringPtr::SetAllocated would move the
+ // string into a fresh copy and delete the owner's one.
+ dst->set_be_exec_version(src->be_exec_version());
+ dst->set_compressed(src->compressed());
+ dst->set_compression_type(src->compression_type());
+ dst->set_uncompressed_size(src->uncompressed_size());
+ dst->set_allocated_column_values(src->mutable_column_values());
Review Comment:
Fixed in a8061f1e2c3. The SHARED borrow now takes the payload pointer through the const accessor (`const_cast<std::string*>(&src->column_values())`) guarded by `DCHECK(src->has_column_values())`, so nothing on the shared holder is written any more. The invariant holds because the sink only enqueues a holder after `Block::serialize` succeeded, which always sets `column_values`; `ArenaStringPtr::Get()` and `UnsafeMutablePointer()` return the same heap pointer, so the borrow/release pairing is unchanged. This also removes the same-value has-bit write that the multi-block broadcast path already had on master.
##########
be/src/util/proto_util.h:
##########
@@ -36,26 +42,128 @@ namespace doris {
// 2G: In the default "baidu_std" brpcd, upper limit of the request and attachment length is 2G.
constexpr size_t MIN_HTTP_BRPC_SIZE = (1ULL << 31);
-// Embed column_values and brpc request serialization string in controller attachment.
+// Attachment layout shared by every http send path:
+//
+// [int64 request_size][serialized request][int64 payload_size][payload]...
+//
+// The serialized request carries everything but the block payloads (PBlock::column_values).
+// One payload section follows per payload block of the request, in block order, see
+// payload_blocks(). A request with a single block therefore keeps the layout older
+// receivers expect; receivers older than this layout read exactly one payload into the
+// singular `block`, so a multi-payload attachment (repeated `blocks`) is only understood by
+// receivers that carry it. Such requests could not be sent at all before (plain brpc rejected
+// them for exceeding its size limit), so no working mixed-version case is lost.
+
+// The blocks whose column_values travel as attachment payloads. A generic request carries one
+// block; PTransmitDataParams carries either the singular `block` or the repeated `blocks`.
+template <typename Params>
+std::vector<PBlock*> payload_blocks(Params* brpc_request) {
+ return {brpc_request->mutable_block()};
+}
+
+inline std::vector<PBlock*> payload_blocks(PTransmitDataParams* brpc_request) {
+ if (brpc_request->has_block()) {
+ DCHECK(brpc_request->blocks().empty());
+ return {brpc_request->mutable_block()};
+ }
+ std::vector<PBlock*> blocks;
+ blocks.reserve(brpc_request->blocks_size());
+ for (auto& block : *brpc_request->mutable_blocks()) {
+ blocks.push_back(&block);
+ }
+ return blocks;
+}
+
+// Append one `[int64 size][bytes]` section to the attachment.
+inline Status attachment_append_section(butil::IOBuf* attachment, const std::string& data) {
+ int64_t data_size = data.size();
+ attachment->append(&data_size, sizeof(data_size));
+ try {
+ attachment->append(data);
+ } catch (...) {
+ LOG(WARNING) << "Try to alloc " << data_size
+ << " bytes for append data to attachment failed. ";
+ return Status::MemoryAllocFailed("request embed attachment failed to memcpy {} bytes",
+ data_size);
+ }
+ return Status::OK();
+}
+
+// Embed `brpc_request` followed by `payloads`, one section each, in the controller attachment.
+template <typename Params, typename Closure>
+Status request_embed_attachmentv2(Params* brpc_request,
+ const std::vector<const std::string*>& payloads,
+ std::unique_ptr<Closure>& closure) {
+ std::string req_str;
+ if (!brpc_request->SerializeToString(&req_str)) {
+ return Status::InternalError("failed to serialize the request");
+ }
+ butil::IOBuf attachment;
+ RETURN_IF_ERROR(attachment_append_section(&attachment, req_str));
+ for (const auto* payload : payloads) {
+ RETURN_IF_ERROR(attachment_append_section(&attachment, *payload));
+ }
+ closure->cntl_->request_attachment().swap(attachment);
+ return Status::OK();
+}
+
+template <typename Params, typename Closure>
+Status request_embed_attachmentv2(Params* brpc_request, const std::string& data,
+ std::unique_ptr<Closure>& closure) {
+ return request_embed_attachmentv2(brpc_request, std::vector<const std::string*> {&data},
+ closure);
+}
+
+// Embed a request that owns its block: column_values is moved into the attachment and the
+// request keeps the field present but empty.
template <typename Params, typename Closure>
Status request_embed_attachment_contain_blockv2(Params* brpc_request,
std::unique_ptr<Closure>& closure) {
- std::string column_values = std::move(*brpc_request->mutable_block()->mutable_column_values());
- brpc_request->mutable_block()->mutable_column_values()->clear();
+ auto* block = brpc_request->mutable_block();
+ std::string column_values = std::move(*block->mutable_column_values());
+ block->mutable_column_values()->clear();
return request_embed_attachmentv2(brpc_request, column_values, closure);
}
+// Embed a request whose blocks only borrow their column_values (see
+// ExchangeSinkBuffer / borrow_block()). Each payload is detached from the request while the
+// request is serialized, appended by reference and attached back, so the owner of the payload
+// is never read through the request's presence bits nor mutated: the same owner may be
+// serialized by other channels at the same time.
+template <typename Closure>
+Status request_embed_attachment_borrowed_blocksv2(PTransmitDataParams* brpc_request,
+ std::unique_ptr<Closure>& closure) {
+ std::vector<PBlock*> blocks = payload_blocks(brpc_request);
+ std::vector<std::string*> payloads;
+ payloads.reserve(blocks.size());
+ for (auto* block : blocks) {
+ payloads.push_back(block->release_column_values());
+ DCHECK(payloads.back() != nullptr);
+ // Keep the field present on the wire: receivers that take the payload slot before
+ // parsing the request rely on the serialized field to restore its presence bit.
+ block->set_column_values("");
+ }
+ Defer attach_back([&]() {
Review Comment:
Fixed in a8061f1e2c3. `Defer attach_back` is registered before the first `release_column_values()` and iterates `payloads.size()`, so a placeholder allocation that throws midway attaches every payload detached so far back to its block; `set_allocated_column_values()` frees the placeholder it replaces, so nothing leaks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6139439629
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6140419227
<details>
<summary>ClickBench: <b>Total hot run time: 24.59 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a8061f1e2c3f5727e29678cf9157fa670a52e4f6, data reload: false
query1 0.01 0.01 0.00
query2 0.10 0.05 0.05
query3 0.26 0.12 0.11
query4 1.61 0.12 0.12
query5 0.25 0.25 0.25
query6 1.17 0.69 0.68
query7 0.04 0.01 0.00
query8 0.05 0.04 0.04
query9 0.42 0.35 0.35
query10 0.58 0.60 0.61
query11 0.20 0.14 0.14
query12 0.18 0.15 0.14
query13 0.50 0.50 0.48
query14 0.97 0.97 0.96
query15 0.63 0.60 0.61
query16 0.34 0.34 0.33
query17 1.12 1.12 1.13
query18 0.25 0.20 0.20
query19 2.16 2.00 1.96
query20 0.02 0.01 0.01
query21 15.55 0.24 0.15
query22 4.56 0.06 0.06
query23 16.22 0.31 0.12
query24 2.86 0.64 0.46
query25 0.13 0.06 0.05
query26 1.52 0.25 0.15
query27 0.04 0.03 0.03
query28 1.90 0.63 0.40
query29 12.49 4.24 3.30
query30 0.28 0.13 0.14
query31 2.77 0.59 0.33
query32 3.22 0.61 0.53
query33 3.16 3.24 3.25
query34 15.48 3.96 3.38
query35 3.39 3.32 3.36
query36 0.55 0.45 0.46
query37 0.10 0.06 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.17 0.16 0.15
query41 0.08 0.02 0.02
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95.5 s
Total hot run time: 24.59 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4452197709
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4621587409
<details>
<summary>TPC-DS: <b>Total hot run time: 169050 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5671323e79e6951c90a481990d512052023b5ca0, data reload: false
query5 4336 643 474 474
query6 441 196 188 188
query7 4816 573 309 309
query8 382 212 206 206
query9 8768 4001 4001 4001
query10 447 341 256 256
query11 5966 2388 2118 2118
query12 162 104 124 104
query13 1246 595 423 423
query14 6423 5377 5019 5019
query14_1 4339 4345 4328 4328
query15 209 201 173 173
query16 994 453 346 346
query17 953 676 575 575
query18 2454 472 333 333
query19 198 182 139 139
query20 106 106 110 106
query21 357 143 120 120
query22 13720 13631 13594 13594
query23 17377 16585 16153 16153
query23_1 16364 16299 16324 16299
query24 7644 1791 1309 1309
query24_1 1318 1301 1342 1301
query25 600 490 415 415
query26 1309 311 178 178
query27 2686 552 359 359
query28 4559 2057 2039 2039
query29 1098 637 508 508
query30 318 238 200 200
query31 1105 1073 973 973
query32 124 65 63 63
query33 540 338 272 272
query34 1238 1116 660 660
query35 751 810 681 681
query36 1380 1412 1263 1263
query37 162 109 94 94
query38 3217 3141 3002 3002
query39 954 915 900 900
query39_1 877 883 894 883
query40 242 126 107 107
query41 75 74 73 73
query42 100 97 101 97
query43 322 327 286 286
query44
query45 207 197 184 184
query46 1087 1228 792 792
query47 2346 2423 2272 2272
query48 392 433 309 309
query49 640 489 366 366
query50 997 376 264 264
query51 4332 4311 4251 4251
query52 92 93 80 80
query53 272 289 200 200
query54 290 234 218 218
query55 80 78 73 73
query56 259 236 243 236
query57 1439 1420 1322 1322
query58 269 232 230 230
query59 1610 1735 1451 1451
query60 294 249 225 225
query61 157 158 162 158
query62 685 653 593 593
query63 225 191 182 182
query64 2584 785 631 631
query65
query66 1788 446 340 340
query67 29835 29737 29466 29466
query68
query69 414 302 256 256
query70 1005 937 905 905
query71 305 229 209 209
query72 3086 2755 2427 2427
query73 869 807 447 447
query74 5135 4902 4743 4743
query75 2671 2597 2242 2242
query76 2307 1170 793 793
query77 383 361 276 276
query78 12364 12496 11828 11828
query79 1454 1037 768 768
query80 585 474 394 394
query81 453 276 241 241
query82 845 155 124 124
query83 362 276 249 249
query84 313 144 111 111
query85 935 538 452 452
query86 395 289 283 283
query87 3388 3304 3134 3134
query88 3647 2820 2685 2685
query89 451 378 335 335
query90 1874 184 174 174
query91 179 169 146 146
query92 62 64 63 63
query93 1433 1390 889 889
query94 543 365 312 312
query95 674 474 348 348
query96 1026 828 365 365
query97 2722 2738 2574 2574
query98 212 212 202 202
query99 1187 1180 1053 1053
Total cold run time: 252314 ms
Total hot run time: 169050 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6081762951
<details>
<summary>ClickBench: <b>Total hot run time: 24.42 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8d44f49991e4bea859b3bdcb1d2f342c3a507aea, data reload: false
query1 0.01 0.01 0.00
query2 0.09 0.04 0.05
query3 0.27 0.12 0.12
query4 1.61 0.11 0.12
query5 0.26 0.26 0.25
query6 1.17 0.69 0.67
query7 0.04 0.01 0.01
query8 0.05 0.04 0.03
query9 0.42 0.35 0.35
query10 0.56 0.57 0.56
query11 0.21 0.14 0.14
query12 0.18 0.14 0.14
query13 0.50 0.49 0.49
query14 0.99 0.96 0.95
query15 0.63 0.62 0.60
query16 0.35 0.35 0.31
query17 1.11 1.14 1.15
query18 0.22 0.20 0.21
query19 2.13 1.99 2.00
query20 0.02 0.02 0.01
query21 15.51 0.27 0.14
query22 4.45 0.05 0.06
query23 16.26 0.34 0.13
query24 2.91 0.73 0.47
query25 0.12 0.04 0.05
query26 1.79 0.26 0.15
query27 0.04 0.04 0.03
query28 2.08 0.56 0.41
query29 12.52 4.24 3.24
query30 0.28 0.14 0.13
query31 2.79 0.59 0.33
query32 3.22 0.62 0.52
query33 3.18 3.22 3.44
query34 15.53 4.02 3.37
query35 3.35 3.30 3.35
query36 0.56 0.43 0.43
query37 0.09 0.06 0.07
query38 0.06 0.03 0.03
query39 0.04 0.03 0.03
query40 0.18 0.15 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.04
query43 0.05 0.04 0.03
Total cold run time: 95.96 s
Total hot run time: 24.42 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6082128949
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6191725842
<details>
<summary>TPC-H: <b>Total hot run time: 29640 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e8fd6bc4b1a1badf954c57536ee02456b4d4da8, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17612 5704 5581 5581
q2 2111 314 271 271
q3 10227 1381 840 840
q4 4687 469 349 349
q5 7446 792 543 543
q6 190 192 157 157
q7 774 788 600 600
q8 9259 1314 1101 1101
q9 5670 4258 4250 4250
q10 6814 1318 1033 1033
q11 441 255 241 241
q12 633 397 291 291
q13 18072 3653 3003 3003
q14 281 270 245 245
q15 q16 743 727 693 693
q17 1171 735 565 565
q18 7119 6230 6195 6195
q19 1096 963 623 623
q20 405 336 227 227
q21 5231 2529 2589 2529
q22 405 368 303 303
Total cold run time: 100387 ms
Total hot run time: 29640 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6516 6499 6436 6436
q2 688 601 569 569
q3 5571 5705 5279 5279
q4 2007 2457 1604 1604
q5 5699 5587 5611 5587
q6 254 195 158 158
q7 2206 2038 1931 1931
q8 3178 2755 2738 2738
q9 8133 7926 7539 7539
q10 3665 3580 3222 3222
q11 591 410 391 391
q12 665 694 505 505
q13 3217 3579 3037 3037
q14 278 290 273 273
q15 q16 670 684 615 615
q17 7823 6942 6864 6864
q18 13137 12321 12994 12321
q19 935 786 786 786
q20 2206 2164 1956 1956
q21 5741 4705 4830 4705
q22 505 459 436 436
Total cold run time: 73685 ms
Total hot run time: 66952 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6196058626
/review-light
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6207342907
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63258:
URL: https://github.com/apache/doris/pull/63258#discussion_r4269698502
##########
be/src/exec/operator/exchange_sink_buffer.cpp:
##########
@@ -49,6 +49,56 @@
namespace doris {
+void borrow_block(PBlock* src, PBlock* dst, BlockSource source) {
+ if (source == BlockSource::OWNED) {
Review Comment:
message PColumnMeta {
message Decimal {
optional uint32 precision = 1;
optional uint32 scale = 2;
}
optional string name = 1 [default = ""];
optional PGenericType.TypeId type = 2 [default = UNKNOWN];
optional bool is_nullable = 3 [default = false];
optional Decimal decimal_param = 4;
repeated PColumnMeta children = 5;
optional bool result_is_nullable = 6;
optional string function_name = 7;
optional int32 be_exec_version = 8;
optional segment_v2.ColumnPathInfo column_path = 9;
optional int32 variant_max_subcolumns_count = 10 [default = 0];
optional bool variant_enable_doc_mode = 11 [default = false];
optional bool variant_is_v2 = 12 [default = false];
}
message PBlock {
repeated PColumnMeta column_metas = 1;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4458605519
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4620335813
# BE UT Coverage Report
Increment line coverage `61.29% (38/62)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/5671323e79e6951c90a481990d512052023b5ca0_5671323e79e6951c90a481990d512052023b5ca0/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/5671323e79e6951c90a481990d512052023b5ca0_5671323e79e6951c90a481990d512052023b5ca0/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.93% (21056/39042) |
| Line Coverage | 37.59% (200179/532486) |
| Region Coverage | 33.67% (156971/466209) |
| Branch Coverage | 34.62% (68666/198319) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6074448655
<details>
<summary>TPC-DS: <b>Total hot run time: 151832 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8592c344d2515cdd06e3b8fade134b93dd79915e, data reload: false
query5 4306 568 443 443
query6 429 207 198 198
query7 4800 434 217 217
query8 324 185 171 171
query9 8773 3937 4067 3937
query10 465 305 264 264
query11 5839 3450 3185 3185
query12 155 89 88 88
query13 1262 451 328 328
query14 6520 4773 4483 4483
query14_1 4221 4251 4210 4210
query15 208 204 188 188
query16 971 456 420 420
query17 863 654 533 533
query18 2424 439 317 317
query19 197 156 163 156
query20 83 80 79 79
query21 212 133 110 110
query22 13026 12946 12741 12741
query23 13105 12571 12204 12204
query23_1 12183 12359 12268 12268
query24 7132 825 461 461
query24_1 464 467 470 467
query25 535 418 358 358
query26 1309 260 141 141
query27 2740 436 272 272
query28 5158 1946 1939 1939
query29 1620 621 471 471
query30 309 234 188 188
query31 890 779 653 653
query32 145 91 95 91
query33 512 313 256 256
query34 919 861 506 506
query35 765 810 662 662
query36 842 856 781 781
query37 135 106 93 93
query38 1825 1776 1663 1663
query39 709 713 692 692
query39_1 649 664 674 664
query40 226 121 100 100
query41 72 70 70 70
query42 84 83 85 83
query43 364 374 328 328
query44 1303 693 687 687
query45 180 190 161 161
query46 829 961 573 573
query47 3018 3012 2953 2953
query48 291 299 211 211
query49 569 396 290 290
query50 668 274 213 213
query51 10161 10389 10071 10071
query52 80 82 71 71
query53 182 211 161 161
query54 249 187 182 182
query55 79 64 63 63
query56 233 225 214 214
query57 1601 1595 1560 1560
query58 284 257 254 254
query59 2363 2363 2099 2099
query60 285 243 229 229
query61 145 154 151 151
query62 396 339 284 284
query63 192 159 165 159
query64 2714 968 786 786
query65 3422 3361 3360 3360
query66 1782 407 292 292
query67 20307 20281 20226 20226
query68 3064 985 603 603
query69 382 286 236 236
query70 915 806 825 806
query71 307 231 212 212
query72 2752 2594 2290 2290
query73 518 525 288 288
query74 4562 4504 4273 4273
query75 2301 2298 1958 1958
query76 2286 1005 583 583
query77 358 380 294 294
query78 8992 8991 8500 8500
query79 1037 902 521 521
query80 563 432 337 337
query81 539 326 274 274
query82 586 132 112 112
query83 307 201 177 177
query84 328 122 107 107
query85 822 419 375 375
query86 323 242 234 234
query87 2010 1948 1824 1824
query88 3570 2655 2649 2649
query89 350 298 261 261
query90 1932 178 180 178
query91 157 148 126 126
query92 94 77 86 77
query93 995 987 556 556
query94 438 299 284 284
query95 606 398 314 314
query96 669 510 226 226
query97 2432 2417 2317 2317
query98 153 149 142 142
query99 790 763 669 669
Total cold run time: 233991 ms
Total hot run time: 151832 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63258:
URL: https://github.com/apache/doris/pull/63258#discussion_r4226535001
##########
be/src/storage/rowset/rowset_meta.cpp:
##########
@@ -300,6 +301,9 @@ bool RowsetMeta::_deserialize_from_pb(std::string_view value) {
_rowset_meta_pb.inverted_index_storage_format());
}
set_tablet_schema(schema_pb);
+ // The schema has been materialized into TabletSchemaCache by set_tablet_schema(). Drop the
+ // protobuf-owned copy from `_rowset_meta_pb` to avoid holding the large schema twice; passing
+ // nullptr intentionally deletes the current protobuf submessage.
_rowset_meta_pb.set_allocated_tablet_schema(nullptr);
Review Comment:
[P2] Rebuild the tablet schema before exporting rowset JSON. `_deserialize_from_pb()` now clears `_rowset_meta_pb.tablet_schema`, but `json_rowset_meta()` passes that same protobuf directly to `ProtoMessageToJson`. After `init(serialized_pb)` on a rowset with a schema, JSON export silently omits `tablet_schema`; loading that JSON cannot recover the schema. Serialize a temporary `get_rowset_pb()` (or reconstruct the schema here) before JSON conversion, and cover the binary-to-JSON round trip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63258:
URL: https://github.com/apache/doris/pull/63258#discussion_r4281754308
##########
be/src/exec/operator/exchange_sink_buffer.cpp:
##########
@@ -49,6 +49,56 @@
namespace doris {
+void borrow_block(PBlock* src, PBlock* dst, BlockSource source) {
+ if (source == BlockSource::OWNED) {
Review Comment:
已按建议简化：column_metas 统一改为 CopyFrom，去掉 OWNED/SHARED 的区分与 BlockSource 枚举；owner 的 column_metas 不再被移走，只有 column_values 仍是借用的。见 963ade7。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6081675598
<details>
<summary>TPC-DS: <b>Total hot run time: 153097 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8d44f49991e4bea859b3bdcb1d2f342c3a507aea, data reload: false
query5 4333 650 483 483
query6 446 233 218 218
query7 4807 431 233 233
query8 333 201 184 184
query9 8792 4000 4022 4000
query10 491 341 278 278
query11 6015 3451 3224 3224
query12 172 97 90 90
query13 1279 454 347 347
query14 6582 4768 4530 4530
query14_1 4318 4198 4254 4198
query15 213 205 184 184
query16 959 447 446 446
query17 903 684 561 561
query18 2428 441 325 325
query19 207 180 142 142
query20 82 82 81 81
query21 212 131 113 113
query22 13050 12967 12803 12803
query23 13099 12472 12084 12084
query23_1 12165 12183 12031 12031
query24 7095 828 476 476
query24_1 474 472 485 472
query25 556 464 378 378
query26 1270 260 145 145
query27 2765 443 272 272
query28 4622 1929 1934 1929
query29 1576 642 487 487
query30 311 220 195 195
query31 907 773 672 672
query32 155 98 101 98
query33 540 330 259 259
query34 937 846 519 519
query35 764 778 670 670
query36 832 815 743 743
query37 134 107 96 96
query38 1801 1737 1722 1722
query39 724 713 677 677
query39_1 696 644 688 644
query40 231 128 107 107
query41 92 91 93 91
query42 101 98 95 95
query43 372 385 328 328
query44 1308 708 701 701
query45 191 185 172 172
query46 849 960 577 577
query47 2963 3010 2764 2764
query48 309 318 232 232
query49 599 416 319 319
query50 676 293 227 227
query51 10299 10197 10435 10197
query52 85 85 77 77
query53 203 221 161 161
query54 262 215 202 202
query55 78 73 70 70
query56 274 256 262 256
query57 1645 1567 1589 1567
query58 307 295 274 274
query59 2386 2402 2145 2145
query60 339 270 257 257
query61 204 211 206 206
query62 411 356 302 302
query63 212 170 173 170
query64 3084 1249 991 991
query65 3411 3364 3376 3364
query66 1826 447 361 361
query67 20101 20064 19902 19902
query68 3053 974 614 614
query69 399 301 259 259
query70 933 819 821 819
query71 307 237 215 215
query72 3013 2784 2581 2581
query73 512 525 292 292
query74 4605 4463 4326 4326
query75 2356 2314 1969 1969
query76 2304 1012 606 606
query77 373 408 312 312
query78 9225 8972 8477 8477
query79 975 874 548 548
query80 573 488 382 382
query81 547 331 285 285
query82 588 136 115 115
query83 323 207 232 207
query84 331 125 112 112
query85 903 533 467 467
query86 320 242 236 236
query87 2012 1945 1849 1849
query88 3581 2697 2687 2687
query89 345 304 269 269
query90 1900 191 182 182
query91 182 173 149 149
query92 98 94 93 93
query93 988 975 556 556
query94 469 330 307 307
query95 626 410 344 344
query96 663 501 241 241
query97 2450 2423 2300 2300
query98 162 150 150 150
query99 778 770 659 659
Total cold run time: 235537 ms
Total hot run time: 153097 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6122164289
Pushed c8b95c634a7, which replaces the previous approach (metadata-only request copy + a second http helper for the shared broadcast block) with one ownership rule for every exchange request:
- `borrow_block()` / `release_borrowed_blocks()` fill and release the request the same way for unicast and broadcast, single-block and multi-block sends; the request never owns block data, so the http attachment path never has to know who owns the payload and never touches the shared `BroadcastPBlockHolder` block.
- The attachment carries one payload per block, so multi-block requests (the default `exchange_multi_blocks_byte_size`) can take the http path too instead of failing on the brpc size limit.
- The receiver parses the request before taking the `column_values` slots, so presence no longer depends on an empty field on the wire; it also validates the attachment framing and `tablet_writer_add_block_by_http` now answers the rpc when extraction fails.
The PR description has been rewritten accordingly. @HappenLee since the mechanism changed after your approval, a second look would be appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6195255647
# BE Regression && UT Coverage Report
Increment line coverage `46.94% (46/98)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63258_6e8fd6bc4b1a1badf954c57536ee02456b4d4da8_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63258_6e8fd6bc4b1a1badf954c57536ee02456b4d4da8_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.09% (35723/46341) |
| Line Coverage | 62.39% (404985/649162) |
| Region Coverage | 58.77% (341965/581836) |
| Branch Coverage | 59.68% (157772/264364) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6079915424
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6084168206
# BE Regression && UT Coverage Report
Increment line coverage `7.14% (1/14)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63258_8d44f49991e4bea859b3bdcb1d2f342c3a507aea_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63258_8d44f49991e4bea859b3bdcb1d2f342c3a507aea_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.79% (35255/45913) |
| Line Coverage | 61.99% (398634/643053) |
| Region Coverage | 58.44% (336651/576110) |
| Branch Coverage | 59.30% (154795/261034) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6136707854
Amended to 8a85aea5c7c (replaces 4494c5ef516): helper consolidation only, no behaviour change. The `[int64 size][bytes]` attachment framing is now written by a single `attachment_append_section()` and read by a single `attachment_read_section()`; `request_embed_attachmentv2` takes the payload list and both the single-payload and the borrowed variants delegate to it; the two `_send_rpc` branches share one `borrow_blocks()` helper. Build, UTs and the regression suite re-run green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63258:
URL: https://github.com/apache/doris/pull/63258#discussion_r4269690986
##########
be/src/exec/operator/exchange_sink_buffer.cpp:
##########
@@ -49,6 +49,56 @@
namespace doris {
+void borrow_block(PBlock* src, PBlock* dst, BlockSource source) {
+ if (source == BlockSource::OWNED) {
Review Comment:
这个meta 感觉不会有多大吧？直接都去copy了会有影响吗？ 代码逻辑是不是简单一些？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6195171021
# BE UT Coverage Report
Increment line coverage `83.67% (82/98)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6e8fd6bc4b1a1badf954c57536ee02456b4d4da8_6e8fd6bc4b1a1badf954c57536ee02456b4d4da8/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6e8fd6bc4b1a1badf954c57536ee02456b4d4da8_6e8fd6bc4b1a1badf954c57536ee02456b4d4da8/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 66.24% (31718/47880) |
| Line Coverage | 51.07% (335337/656577) |
| Region Coverage | 46.36% (270908/584302) |
| Branch Coverage | 47.99% (127257/265197) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6209747052
# BE Regression && UT Coverage Report
Increment line coverage `46.94% (46/98)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63258_1483f43b2bc9721b2a636a9d7805490469bda5d8_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63258_1483f43b2bc9721b2a636a9d7805490469bda5d8_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.17% (35719/46287) |
| Line Coverage | 62.39% (403026/645942) |
| Region Coverage | 58.74% (339710/578310) |
| Branch Coverage | 59.72% (156981/262879) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4452518653
<details>
<summary>TPC-DS: <b>Total hot run time: 171837 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a5ae6300f8122b1ec603a2727913f39eac342273, data reload: false
query5 4303 659 520 520
query6 344 216 210 210
query7 4252 559 300 300
query8 325 252 221 221
query9 8805 4024 4014 4014
query10 451 335 288 288
query11 5791 2347 2213 2213
query12 186 140 133 133
query13 1264 615 445 445
query14 5928 5376 5052 5052
query14_1 4318 4416 4348 4348
query15 209 207 184 184
query16 1045 490 444 444
query17 1145 768 646 646
query18 2604 507 375 375
query19 224 212 166 166
query20 137 132 129 129
query21 221 146 118 118
query22 13617 13969 14580 13969
query23 17126 16615 16224 16224
query23_1 16156 16271 16284 16271
query24 7395 1721 1324 1324
query24_1 1391 1360 1379 1360
query25 582 474 414 414
query26 1305 309 163 163
query27 2705 612 327 327
query28 4347 1929 1943 1929
query29 985 666 520 520
query30 290 237 194 194
query31 1102 1059 928 928
query32 81 74 69 69
query33 541 345 285 285
query34 1143 1142 627 627
query35 781 781 662 662
query36 1324 1333 1167 1167
query37 148 101 85 85
query38 3161 3113 3073 3073
query39 926 903 892 892
query39_1 869 862 880 862
query40 237 159 132 132
query41 63 60 61 60
query42 106 112 107 107
query43 320 327 282 282
query44
query45 209 198 193 193
query46 1035 1175 721 721
query47 2288 2243 2235 2235
query48 395 423 299 299
query49 632 524 416 416
query50 691 284 216 216
query51 4264 4297 4178 4178
query52 103 103 94 94
query53 254 276 200 200
query54 311 275 267 267
query55 96 91 85 85
query56 300 307 300 300
query57 1405 1375 1260 1260
query58 286 269 262 262
query59 1521 1612 1375 1375
query60 334 339 323 323
query61 164 157 165 157
query62 670 616 545 545
query63 237 204 212 204
query64 2397 835 701 701
query65
query66 1697 520 402 402
query67 29933 29844 29849 29844
query68
query69 452 327 296 296
query70 1037 910 989 910
query71 305 273 275 273
query72 2937 2714 2376 2376
query73 858 754 422 422
query74 5046 4893 4701 4701
query75 2747 2693 2329 2329
query76 2295 1133 778 778
query77 407 424 355 355
query78 12955 12940 12446 12446
query79 1448 933 745 745
query80 673 575 488 488
query81 456 280 238 238
query82 1342 158 122 122
query83 366 278 270 270
query84 256 137 113 113
query85 868 524 451 451
query86 397 347 356 347
query87 3383 3362 3212 3212
query88 3548 2730 2686 2686
query89 447 386 339 339
query90 1897 190 188 188
query91 196 179 152 152
query92 81 80 75 75
query93 960 954 548 548
query94 563 364 279 279
query95 685 381 355 355
query96 1027 810 342 342
query97 2699 2674 2538 2538
query98 239 224 224 224
query99 1138 1111 965 965
Total cold run time: 251955 ms
Total hot run time: 171837 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4451606402
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4451911629
<details>
<summary>TPC-DS: <b>Total hot run time: 170079 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cdde6d13200718593069301b60edc5624d1ea419, data reload: false
query5 4308 647 508 508
query6 329 213 205 205
query7 4267 593 300 300
query8 330 224 217 217
query9 8821 4008 3961 3961
query10 451 336 304 304
query11 5789 2398 2194 2194
query12 185 129 127 127
query13 1275 588 440 440
query14 6452 5340 5077 5077
query14_1 4353 4363 4341 4341
query15 219 207 188 188
query16 997 468 374 374
query17 1170 728 608 608
query18 2723 494 346 346
query19 208 193 164 164
query20 139 130 126 126
query21 214 134 116 116
query22 13562 13541 13309 13309
query23 17092 16318 16059 16059
query23_1 16113 16174 16176 16174
query24 7358 1738 1326 1326
query24_1 1339 1344 1357 1344
query25 563 499 444 444
query26 1320 296 173 173
query27 2702 573 338 338
query28 4426 1969 1950 1950
query29 979 630 509 509
query30 302 238 197 197
query31 1177 1068 941 941
query32 87 77 72 72
query33 556 349 289 289
query34 1187 1150 638 638
query35 755 777 677 677
query36 1325 1307 1123 1123
query37 152 103 88 88
query38 3215 3103 3040 3040
query39 915 934 926 926
query39_1 895 871 864 864
query40 232 158 137 137
query41 81 64 60 60
query42 111 111 106 106
query43 323 323 281 281
query44
query45 217 205 201 201
query46 1082 1175 709 709
query47 2281 2261 2191 2191
query48 383 415 335 335
query49 647 536 413 413
query50 716 281 221 221
query51 4325 4293 4239 4239
query52 113 105 95 95
query53 254 285 199 199
query54 313 272 266 266
query55 95 89 89 89
query56 307 308 304 304
query57 1529 1375 1301 1301
query58 296 274 272 272
query59 1524 1642 1380 1380
query60 340 334 322 322
query61 162 157 158 157
query62 671 644 571 571
query63 252 202 208 202
query64 2365 808 681 681
query65
query66 1696 509 404 404
query67 30027 29941 29866 29866
query68
query69 455 341 289 289
query70 1030 970 920 920
query71 301 263 269 263
query72 2993 2740 2403 2403
query73 817 779 423 423
query74 5091 4893 4748 4748
query75 2777 2694 2346 2346
query76 2314 1134 752 752
query77 428 432 354 354
query78 12842 12900 12361 12361
query79 1489 963 713 713
query80 730 612 529 529
query81 452 277 244 244
query82 1347 165 123 123
query83 367 287 260 260
query84 264 153 121 121
query85 926 585 532 532
query86 399 324 316 316
query87 3506 3347 3200 3200
query88 3553 2677 2646 2646
query89 453 388 340 340
query90 1909 183 181 181
query91 217 165 139 139
query92 76 74 70 70
query93 952 944 559 559
query94 514 331 307 307
query95 654 369 356 356
query96 1013 813 347 347
query97 2701 2667 2549 2549
query98 239 229 227 227
query99 1129 1111 995 995
Total cold run time: 253499 ms
Total hot run time: 170079 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4451571813
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-4458788651
<details>
<summary>TPC-DS: <b>Total hot run time: 170052 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5671323e79e6951c90a481990d512052023b5ca0, data reload: false
query5 4303 679 514 514
query6 344 224 204 204
query7 4209 589 299 299
query8 328 234 219 219
query9 8824 4008 3988 3988
query10 451 333 297 297
query11 5788 2406 2188 2188
query12 179 128 130 128
query13 1281 629 419 419
query14 5951 5378 5053 5053
query14_1 4380 4351 4376 4351
query15 216 213 183 183
query16 1011 412 457 412
query17 990 748 606 606
query18 2467 499 370 370
query19 222 214 191 191
query20 136 133 132 132
query21 216 136 119 119
query22 13634 13601 13382 13382
query23 17232 16469 16014 16014
query23_1 16088 16031 16190 16031
query24 7365 1718 1280 1280
query24_1 1309 1314 1312 1312
query25 550 466 415 415
query26 1314 326 177 177
query27 2673 570 334 334
query28 4434 1967 1945 1945
query29 960 606 523 523
query30 303 243 199 199
query31 1128 1088 945 945
query32 85 75 72 72
query33 516 357 310 310
query34 1147 1128 648 648
query35 764 792 678 678
query36 1331 1354 1187 1187
query37 152 103 89 89
query38 3228 3133 3064 3064
query39 926 931 895 895
query39_1 888 881 860 860
query40 232 151 127 127
query41 67 64 62 62
query42 112 117 110 110
query43 327 326 286 286
query44
query45 216 209 203 203
query46 1082 1181 721 721
query47 2323 2374 2162 2162
query48 395 415 287 287
query49 643 505 394 394
query50 964 365 259 259
query51 4369 4365 4209 4209
query52 106 107 96 96
query53 259 282 207 207
query54 317 279 254 254
query55 95 97 84 84
query56 307 332 307 307
query57 1398 1393 1326 1326
query58 303 277 272 272
query59 1576 1645 1362 1362
query60 321 336 297 297
query61 160 152 160 152
query62 675 629 561 561
query63 250 212 198 198
query64 2393 811 674 674
query65
query66 1705 505 379 379
query67 30093 29898 29838 29838
query68
query69 470 339 309 309
query70 973 997 992 992
query71 320 282 264 264
query72 3057 2719 2465 2465
query73 872 763 445 445
query74 5071 4918 4742 4742
query75 2660 2632 2252 2252
query76 2270 1156 785 785
query77 420 417 318 318
query78 12123 12010 11611 11611
query79 1272 1017 725 725
query80 590 564 453 453
query81 465 282 240 240
query82 228 153 121 121
query83 268 271 250 250
query84 293 141 115 115
query85 897 617 454 454
query86 395 353 314 314
query87 3385 3389 3244 3244
query88 3485 2656 2631 2631
query89 423 388 337 337
query90 2211 182 182 182
query91 194 165 138 138
query92 85 77 74 74
query93 1391 1435 918 918
query94 556 351 295 295
query95 675 491 360 360
query96 993 774 334 334
query97 2678 2693 2576 2576
query98 235 231 227 227
query99 1127 1105 973 973
Total cold run time: 251112 ms
Total hot run time: 170052 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6074497916
<details>
<summary>ClickBench: <b>Total hot run time: 24.55 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8592c344d2515cdd06e3b8fade134b93dd79915e, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.12 0.12
query4 1.60 0.12 0.13
query5 0.28 0.22 0.25
query6 1.17 0.68 0.69
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.43 0.35 0.36
query10 0.59 0.58 0.61
query11 0.21 0.14 0.15
query12 0.18 0.15 0.14
query13 0.50 0.49 0.50
query14 0.97 0.98 0.99
query15 0.64 0.60 0.62
query16 0.36 0.34 0.36
query17 1.13 1.14 1.15
query18 0.23 0.21 0.21
query19 2.10 1.95 2.03
query20 0.02 0.01 0.01
query21 15.51 0.27 0.16
query22 4.48 0.05 0.06
query23 16.24 0.32 0.13
query24 3.13 0.59 0.46
query25 0.11 0.05 0.05
query26 1.42 0.24 0.15
query27 0.04 0.03 0.04
query28 1.90 0.60 0.42
query29 12.53 4.06 3.24
query30 0.29 0.14 0.14
query31 2.78 0.60 0.33
query32 3.23 0.61 0.53
query33 3.19 3.17 3.18
query34 15.51 4.02 3.37
query35 3.34 3.34 3.34
query36 0.54 0.43 0.43
query37 0.10 0.07 0.07
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95.64 s
Total hot run time: 24.55 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6074012469
/review-light
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6074348833
<details>
<summary>TPC-H: <b>Total hot run time: 28763 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8592c344d2515cdd06e3b8fade134b93dd79915e, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17606 5587 5538 5538
q2 2118 310 265 265
q3 10238 1385 831 831
q4 4679 479 344 344
q5 7436 791 547 547
q6 192 192 154 154
q7 752 798 621 621
q8 9248 1356 1036 1036
q9 5674 4437 4426 4426
q10 6810 1304 1043 1043
q11 448 257 241 241
q12 631 405 289 289
q13 18038 3051 2395 2395
q14 278 270 254 254
q15 q16 741 724 671 671
q17 1179 740 581 581
q18 7029 6240 6182 6182
q19 1104 970 604 604
q20 383 337 225 225
q21 4451 2234 2407 2234
q22 359 310 282 282
Total cold run time: 99394 ms
Total hot run time: 28763 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6166 6125 6135 6125
q2 684 567 542 542
q3 5256 5389 4978 4978
q4 2037 2149 1558 1558
q5 5298 5200 5873 5200
q6 251 196 156 156
q7 2331 2072 1867 1867
q8 3157 2783 2769 2769
q9 8238 8305 8350 8305
q10 3761 3704 3379 3379
q11 622 453 392 392
q12 709 751 520 520
q13 2909 3304 2520 2520
q14 335 309 277 277
q15 q16 730 743 676 676
q17 8452 7603 7300 7300
q18 13179 12278 13065 12278
q19 937 793 785 785
q20 2177 2166 1935 1935
q21 5817 4743 4817 4743
q22 502 460 402 402
Total cold run time: 73548 ms
Total hot run time: 66707 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6076061935
run nonConcurrent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63258:
URL: https://github.com/apache/doris/pull/63258#discussion_r4254216551
##########
be/src/exec/operator/exchange_sink_buffer.cpp:
##########
@@ -49,6 +49,52 @@
namespace doris {
+void borrow_block(PBlock* src, PBlock* dst, BlockSource source) {
+ if (source == BlockSource::OWNED) {
+ // O(1) pointer swap; the owner is destroyed right after the send and takes the request's
+ // cleared metas with it.
+ dst->mutable_column_metas()->Swap(src->mutable_column_metas());
+ } else {
+ dst->mutable_column_metas()->CopyFrom(src->column_metas());
+ }
+ // set_allocated_column_values() stores the owner's std::string pointer as is; the thirdparty
+ // protobuf is built without NDEBUG checks, otherwise ArenaStringPtr::SetAllocated would move the
+ // string into a fresh copy and delete the owner's one.
+ dst->set_be_exec_version(src->be_exec_version());
+ dst->set_compressed(src->compressed());
+ dst->set_compression_type(src->compression_type());
+ dst->set_uncompressed_size(src->uncompressed_size());
+ dst->set_allocated_column_values(src->mutable_column_values());
Review Comment:
[P2] Keep shared-source lookup free of mutable protobuf access
With `exchange_multi_blocks_byte_size = -1`, an unpartitioned exchange can queue one `BroadcastPBlockHolder` for two busy destinations. Their previous RPC completions then call `_send_rpc()` concurrently under different `RpcInstance` mutexes and both reach this call. In pinned protobuf 21.11, the [mutable string accessor](https://github.com/protocolbuffers/protobuf/blob/v21.11/src/google/protobuf/compiler/cpp/string_field.cc#L243-L275) always [writes the non-atomic presence bits](https://github.com/protocolbuffers/protobuf/blob/v21.11/src/google/protobuf/compiler/cpp/field.cc#L269-L279), even when the payload is already present, producing a data race on the shared owner. This is distinct from the earlier HTTP release/restore thread: that payload mutation is fixed, but the old single-block branch only borrowed the whole `PBlock`, and this helper adds source mutation even for small ordinary RPCs. Obtain the already-initialized shared payload through the const `column_values()` acces
sor and keep mutable protobuf operations on the destination request.
##########
be/src/util/proto_util.h:
##########
@@ -36,26 +42,128 @@ namespace doris {
// 2G: In the default "baidu_std" brpcd, upper limit of the request and attachment length is 2G.
constexpr size_t MIN_HTTP_BRPC_SIZE = (1ULL << 31);
-// Embed column_values and brpc request serialization string in controller attachment.
+// Attachment layout shared by every http send path:
+//
+// [int64 request_size][serialized request][int64 payload_size][payload]...
+//
+// The serialized request carries everything but the block payloads (PBlock::column_values).
+// One payload section follows per payload block of the request, in block order, see
+// payload_blocks(). A request with a single block therefore keeps the layout older
+// receivers expect; receivers older than this layout read exactly one payload into the
+// singular `block`, so a multi-payload attachment (repeated `blocks`) is only understood by
+// receivers that carry it. Such requests could not be sent at all before (plain brpc rejected
+// them for exceeding its size limit), so no working mixed-version case is lost.
+
+// The blocks whose column_values travel as attachment payloads. A generic request carries one
+// block; PTransmitDataParams carries either the singular `block` or the repeated `blocks`.
+template <typename Params>
+std::vector<PBlock*> payload_blocks(Params* brpc_request) {
+ return {brpc_request->mutable_block()};
+}
+
+inline std::vector<PBlock*> payload_blocks(PTransmitDataParams* brpc_request) {
+ if (brpc_request->has_block()) {
+ DCHECK(brpc_request->blocks().empty());
+ return {brpc_request->mutable_block()};
+ }
+ std::vector<PBlock*> blocks;
+ blocks.reserve(brpc_request->blocks_size());
+ for (auto& block : *brpc_request->mutable_blocks()) {
+ blocks.push_back(&block);
+ }
+ return blocks;
+}
+
+// Append one `[int64 size][bytes]` section to the attachment.
+inline Status attachment_append_section(butil::IOBuf* attachment, const std::string& data) {
+ int64_t data_size = data.size();
+ attachment->append(&data_size, sizeof(data_size));
+ try {
+ attachment->append(data);
+ } catch (...) {
+ LOG(WARNING) << "Try to alloc " << data_size
+ << " bytes for append data to attachment failed. ";
+ return Status::MemoryAllocFailed("request embed attachment failed to memcpy {} bytes",
+ data_size);
+ }
+ return Status::OK();
+}
+
+// Embed `brpc_request` followed by `payloads`, one section each, in the controller attachment.
+template <typename Params, typename Closure>
+Status request_embed_attachmentv2(Params* brpc_request,
+ const std::vector<const std::string*>& payloads,
+ std::unique_ptr<Closure>& closure) {
+ std::string req_str;
+ if (!brpc_request->SerializeToString(&req_str)) {
+ return Status::InternalError("failed to serialize the request");
+ }
+ butil::IOBuf attachment;
+ RETURN_IF_ERROR(attachment_append_section(&attachment, req_str));
+ for (const auto* payload : payloads) {
+ RETURN_IF_ERROR(attachment_append_section(&attachment, *payload));
+ }
+ closure->cntl_->request_attachment().swap(attachment);
+ return Status::OK();
+}
+
+template <typename Params, typename Closure>
+Status request_embed_attachmentv2(Params* brpc_request, const std::string& data,
+ std::unique_ptr<Closure>& closure) {
+ return request_embed_attachmentv2(brpc_request, std::vector<const std::string*> {&data},
+ closure);
+}
+
+// Embed a request that owns its block: column_values is moved into the attachment and the
+// request keeps the field present but empty.
template <typename Params, typename Closure>
Status request_embed_attachment_contain_blockv2(Params* brpc_request,
std::unique_ptr<Closure>& closure) {
- std::string column_values = std::move(*brpc_request->mutable_block()->mutable_column_values());
- brpc_request->mutable_block()->mutable_column_values()->clear();
+ auto* block = brpc_request->mutable_block();
+ std::string column_values = std::move(*block->mutable_column_values());
+ block->mutable_column_values()->clear();
return request_embed_attachmentv2(brpc_request, column_values, closure);
}
+// Embed a request whose blocks only borrow their column_values (see
+// ExchangeSinkBuffer / borrow_block()). Each payload is detached from the request while the
+// request is serialized, appended by reference and attached back, so the owner of the payload
+// is never read through the request's presence bits nor mutated: the same owner may be
+// serialized by other channels at the same time.
+template <typename Closure>
+Status request_embed_attachment_borrowed_blocksv2(PTransmitDataParams* brpc_request,
+ std::unique_ptr<Closure>& closure) {
+ std::vector<PBlock*> blocks = payload_blocks(brpc_request);
+ std::vector<std::string*> payloads;
+ payloads.reserve(blocks.size());
+ for (auto* block : blocks) {
+ payloads.push_back(block->release_column_values());
+ DCHECK(payloads.back() != nullptr);
+ // Keep the field present on the wire: receivers that take the payload slot before
+ // parsing the request rely on the serialized field to restore its presence bit.
+ block->set_column_values("");
+ }
+ Defer attach_back([&]() {
Review Comment:
[P3] Install the restoration guard before creating placeholders
`release_column_values()` resets each request slot to its default, so the following `set_column_values("")` [allocates a new empty string object](https://github.com/protocolbuffers/protobuf/blob/v21.11/src/google/protobuf/arenastring.cc#L99-L128). In a multi-block HTTP batch, if the first placeholder succeeds and a later allocation throws, this `attach_back` guard has not been constructed. The enclosing `_send_rpc()` guard then calls `release_borrowed_blocks()`, which releases and discards the earlier request-owned placeholder as though it were a borrowed owner string. The original payloads remain safe, but those empty string objects have no remaining owner and leak during unwind. Register restoration before the loop and restore the payloads already detached if a later placeholder allocation fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6076393602
# BE Regression && UT Coverage Report
Increment line coverage `0.00% (0/13)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63258_8592c344d2515cdd06e3b8fade134b93dd79915e_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63258_8592c344d2515cdd06e3b8fade134b93dd79915e_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.54% (35136/45903) |
| Line Coverage | 61.74% (396895/642876) |
| Region Coverage | 58.15% (334814/575746) |
| Branch Coverage | 59.07% (154127/260930) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63258:
URL: https://github.com/apache/doris/pull/63258#discussion_r4229106903
##########
be/src/util/proto_util.h:
##########
@@ -39,9 +40,24 @@ constexpr size_t MIN_HTTP_BRPC_SIZE = (1ULL << 31);
// Embed column_values and brpc request serialization string in controller attachment.
template <typename Params, typename Closure>
Status request_embed_attachment_contain_blockv2(Params* brpc_request,
- std::unique_ptr<Closure>& closure) {
- std::string column_values = std::move(*brpc_request->mutable_block()->mutable_column_values());
- brpc_request->mutable_block()->mutable_column_values()->clear();
+ std::unique_ptr<Closure>& closure,
+ bool restore_column_values = false) {
+ auto* block = brpc_request->mutable_block();
+ if (restore_column_values) {
+ // Some callers borrow block storage from a shared owner. Temporarily detach the large
+ // column_values field so the serialized request stays small, then restore it before
+ // returning so the real owner can still be reused by later sends.
+ auto* column_values = block->release_column_values();
+ DORIS_CHECK(column_values != nullptr);
Review Comment:
The restore mode is gone. Since 27e530b3639 the broadcast HTTP path no longer touches the shared `PBlock`: it builds a metadata-only request copy (`make_http_request_without_column_values`) and passes the holder's `column_values` as read-only attachment data through `transmit_block_httpv2_with_attachment_data`, so concurrent sends of the same holder from different `RpcInstance` callbacks only read it. Resolving as addressed.
##########
be/src/storage/rowset/rowset_meta.cpp:
##########
@@ -300,6 +301,9 @@ bool RowsetMeta::_deserialize_from_pb(std::string_view value) {
_rowset_meta_pb.inverted_index_storage_format());
}
set_tablet_schema(schema_pb);
+ // The schema has been materialized into TabletSchemaCache by set_tablet_schema(). Drop the
+ // protobuf-owned copy from `_rowset_meta_pb` to avoid holding the large schema twice; passing
+ // nullptr intentionally deletes the current protobuf submessage.
_rowset_meta_pb.set_allocated_tablet_schema(nullptr);
Review Comment:
Not changed in this PR. `_deserialize_from_pb()` clearing `tablet_schema` with `set_allocated_tablet_schema(nullptr)` and `json_rowset_meta()` serializing `_rowset_meta_pb` directly both already exist on master (merge base 9a5f86ec82b, `rowset_meta.cpp:303`); this PR only adds a comment above that line. `json_rowset_meta()` also has no production caller under `be/src`. The JSON export behaviour is pre-existing and out of scope for this exchange fix, so I am leaving it as is here.
##########
be/src/exec/operator/exchange_sink_buffer.cpp:
##########
@@ -49,6 +49,49 @@
namespace doris {
+namespace exchange_sink_buffer::detail {
+
+void copy_block_metadata_without_column_values(const PBlock& src, PBlock* dst) {
+ for (int i = 0; i < src.column_metas_size(); ++i) {
+ dst->add_column_metas()->CopyFrom(src.column_metas(i));
+ }
+ dst->set_be_exec_version(src.be_exec_version());
+ dst->set_compressed(src.compressed());
+ dst->set_compression_type(src.compression_type());
+ dst->set_uncompressed_size(src.uncompressed_size());
+}
+
+std::shared_ptr<PTransmitDataParams> make_http_request_without_column_values(
+ const PTransmitDataParams& src) {
+ DORIS_CHECK(src.has_block());
+ DORIS_CHECK(src.blocks_size() == 0);
+ DORIS_CHECK(!src.has_row_batch());
+
+ auto dst = std::make_shared<PTransmitDataParams>();
+ dst->mutable_finst_id()->CopyFrom(src.finst_id());
+ dst->set_node_id(src.node_id());
+ dst->set_sender_id(src.sender_id());
+ dst->set_be_number(src.be_number());
+ dst->set_eos(src.eos());
+ dst->set_packet_seq(src.packet_seq());
+ if (src.has_query_statistics()) {
+ dst->mutable_query_statistics()->CopyFrom(src.query_statistics());
+ }
+ if (src.has_transfer_by_attachment()) {
+ dst->set_transfer_by_attachment(src.transfer_by_attachment());
+ }
+ if (src.has_query_id()) {
+ dst->mutable_query_id()->CopyFrom(src.query_id());
+ }
+ if (src.has_exec_status()) {
+ dst->mutable_exec_status()->CopyFrom(src.exec_status());
+ }
+ copy_block_metadata_without_column_values(src.block(), dst->mutable_block());
Review Comment:
Fixed in 8d44f49991e. `copy_block_metadata_without_column_values()` now marks `column_values` present with an empty value (`dst->set_column_values("")`), the same shape `request_embed_attachment_contain_blockv2()` leaves behind after moving the owned payload out, so the receiver's `ParseFromString()` sets the presence bit and the `PBlock::CopyFrom()` in `VDataStreamMgr::transmit_block()` keeps the extracted bytes.
Tests: added `ExchangeSinkBufferTest.HttpRequestKeepsColumnValuesThroughAttachmentAndCopyFrom` (metadata-only request + `request_embed_attachmentv2` with the borrowed payload -> `attachment_extract_request_contain_block` -> `PBlock::CopyFrom`) and `ProtoUtilTest.AttachmentWithoutColumnValuesPresenceIsDroppedByCopyFrom`, which documents the failure shape; the existing tests now assert presence after extraction and after `CopyFrom`.
Verified end to end on a local 1 FE + 2 BE cluster with the forced-HTTP debug point: without this line the broadcast join fails on the receiver (`Allocator sys memory check failed: Cannot alloc:134217728.00 GB`, from deserializing an empty payload); with it both queries pass.
##########
regression-test/suites/correctness_p0/test_exchange_http_send_borrowed_block.groovy:
##########
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.ClusterOptions
+
+// Forces exchange sinks to use the http-attachment path (normally reserved for
+// requests >= 2G) via a debug point, so that both the unicast (shuffle) and
+// broadcast exchange sink code in proto_util.h/exchange_sink_buffer.cpp that
+// move/borrow column_values into the brpc attachment actually run cross-BE,
+// instead of only through unit tests.
+suite('test_exchange_http_send_borrowed_block', 'docker') {
+ def options = new ClusterOptions()
+ options.beNum = 2
+ options.enableDebugPoints()
+
+ docker(options) {
+ def tbl = 'test_exchange_http_send_borrowed_block_tbl'
+
+ sql "DROP TABLE IF EXISTS ${tbl}"
+ sql """
+ CREATE TABLE ${tbl} (
+ k1 INT NULL,
+ k2 INT NULL
+ )
+ DISTRIBUTED BY HASH(k1) BUCKETS 4
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO ${tbl} VALUES
+ (1, 10), (2, 20), (3, 30), (4, 40), (5, 50), (6, 60), (7, 70), (8, 80)
+ """
+
+ def backends = cluster.getAllBackends()
+ assertTrue(backends.size() == 2)
+ for (def be : backends) {
+ be.enableDebugPoint('proto_util.enable_http_send_block.always_http', null)
+ }
+
+ try {
+ // Shuffle exchange: exercises the unicast attachment path
Review Comment:
Fixed in 8d44f49991e. The suite now runs `SET exchange_multi_blocks_byte_size = -1` before the queries, and after each query asserts that the sum over all BEs of brpc's built-in `rpc_server_<port>_doris_pbackend_service_transmit_block_by_http_count` (read from `/brpc_metrics` via `WarmupMetricsUtils.getBrpcMetric`) increased, so a run cannot pass without reaching `transmit_block_by_http`.
While verifying this I also found that the self join on the distribution column `k1` was planned as a COLOCATE join with no exchange at all, so both queries now join on `k2` (same result rows, `.out` unchanged). On a local 2 BE cluster the counter goes 0 -> 6 after the shuffle join and -> 10 after the broadcast join, and the broadcast query fails there without the presence fix above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6136862328
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve borrowed block data for http send [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63258:
URL: https://github.com/apache/doris/pull/63258#issuecomment-6208371771
<details>
<summary>TPC-DS: <b>Total hot run time: 151463 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1483f43b2bc9721b2a636a9d7805490469bda5d8, data reload: false
query5 4306 594 461 461
query6 426 208 184 184
query7 4805 440 224 224
query8 325 170 171 170
query9 8764 3956 3919 3919
query10 492 318 253 253
query11 5900 3809 3509 3509
query12 145 91 86 86
query13 1255 439 321 321
query14 6102 3977 3759 3759
query14_1 3522 3511 3442 3442
query15 204 200 184 184
query16 961 437 409 409
query17 857 654 542 542
query18 2448 403 295 295
query19 200 153 133 133
query20 83 92 82 82
query21 212 128 108 108
query22 12920 13005 12715 12715
query23 13411 12968 12552 12552
query23_1 12698 12597 12650 12597
query24 7050 840 458 458
query24_1 458 472 460 460
query25 499 387 339 339
query26 1325 257 133 133
query27 2771 437 269 269
query28 4625 1938 1967 1938
query29 1568 579 438 438
query30 300 216 193 193
query31 989 923 801 801
query32 140 91 85 85
query33 509 290 235 235
query34 908 849 511 511
query35 737 773 650 650
query36 832 798 685 685
query37 134 103 89 89
query38 1794 1728 1703 1703
query39 795 812 748 748
query39_1 745 723 751 723
query40 209 122 101 101
query41 64 62 61 61
query42 84 80 82 80
query43 373 371 330 330
query44 1279 689 686 686
query45 180 175 163 163
query46 861 958 561 561
query47 2987 3018 2939 2939
query48 292 313 226 226
query49 599 410 299 299
query50 690 283 215 215
query51 10689 10463 10528 10463
query52 80 79 79 79
query53 189 245 154 154
query54 232 195 183 183
query55 72 64 63 63
query56 223 235 209 209
query57 1597 1563 1534 1534
query58 294 267 256 256
query59 2096 2142 1964 1964
query60 281 233 217 217
query61 147 142 134 134
query62 385 345 286 286
query63 199 165 157 157
query64 2720 915 741 741
query65 3253 3232 3225 3225
query66 1782 415 292 292
query67 20209 20007 19704 19704
query68 3053 991 612 612
query69 394 281 243 243
query70 858 794 791 791
query71 293 226 204 204
query72 2789 2569 2297 2297
query73 520 518 307 307
query74 4532 4448 4310 4310
query75 2348 2277 1928 1928
query76 2324 1019 591 591
query77 350 426 268 268
query78 9827 9632 9139 9139
query79 865 845 509 509
query80 477 427 342 342
query81 508 327 270 270
query82 207 138 111 111
query83 195 194 180 180
query84 286 118 96 96
query85 772 398 374 374
query86 280 245 236 236
query87 2020 1946 1807 1807
query88 3566 2695 2669 2669
query89 345 297 256 256
query90 2116 176 183 176
query91 157 150 122 122
query92 100 91 86 86
query93 975 981 562 562
query94 444 309 277 277
query95 594 398 321 321
query96 655 515 235 235
query97 2478 2447 2339 2339
query98 161 152 145 145
query99 760 772 651 651
Total cold run time: 233348 ms
Total hot run time: 151463 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org