You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wangbo (via GitHub)" <gi...@apache.org> on 2024/09/24 03:48:17 UTC
[PR] [Improvement]Add customStdAllocator for vector/map [doris]
wangbo opened a new pull request, #41193:
URL: https://github.com/apache/doris/pull/41193
## Proposed changes
1 Add customStdAllocator for vector/map.
2 Manually count memory for delete bitmap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #41193:
URL: https://github.com/apache/doris/pull/41193#discussion_r1772560656
##########
be/src/olap/rowid_conversion.h:
##########
@@ -115,6 +121,7 @@ class RowIdConversion {
// value indicates row id of destination segment.
// <UINT32_MAX, UINT32_MAX> indicates current row not exist.
std::vector<std::vector<std::pair<uint32_t, uint32_t>>> _segments_rowid_map;
Review Comment:
Why not replace to MyVector here (just like what you did in SimpleRowidConversion)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2370919686
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2374354987
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2370657218
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #41193:
URL: https://github.com/apache/doris/pull/41193#discussion_r1774372420
##########
be/src/vec/common/custom_allocator.h:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "vec/common/allocator.h"
+#include "vec/common/allocator_fwd.h"
+
+template <class T, typename MemoryAllocator = Allocator<true>>
+class CustomStdAllocator;
+
+template <typename T>
+using DorisVector = std::vector<T, CustomStdAllocator<T>>;
+
+template <typename K, typename T>
+using DorisMap = std::map<K, T, std::less<K>, CustomStdAllocator<std::pair<const K, T>>>;
Review Comment:
https://en.cppreference.com/w/cpp/container/map
template<
class Key,
class T,
class Compare = [std::less](http://en.cppreference.com/w/cpp/utility/functional/less)<Key>,
class Allocator = [std::allocator](http://en.cppreference.com/w/cpp/memory/allocator)<[std::pair](http://en.cppreference.com/w/cpp/utility/pair)<const Key, T>>
> class map;
##########
be/src/olap/rowid_conversion.h:
##########
@@ -33,17 +34,36 @@ namespace doris {
class RowIdConversion {
public:
RowIdConversion() = default;
- ~RowIdConversion() = default;
+ ~RowIdConversion() {
+ if (_row_id_thread_ctx_memtracker_ptr) {
+ SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER(_row_id_thread_ctx_memtracker_ptr);
Review Comment:
这里没必要switch，直接release 就可以
##########
be/src/olap/rowid_conversion.h:
##########
@@ -33,17 +34,36 @@ namespace doris {
class RowIdConversion {
public:
RowIdConversion() = default;
- ~RowIdConversion() = default;
+ ~RowIdConversion() {
+ if (_row_id_thread_ctx_memtracker_ptr) {
+ SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER(_row_id_thread_ctx_memtracker_ptr);
+ RELEASE_THREAD_MEM_TRACKER(_seg_rowid_map_mem_used);
+ }
+ }
// resize segment rowid map to its rows num
void init_segment_map(const RowsetId& src_rowset_id, const std::vector<uint32_t>& num_rows) {
+ size_t mem_used = 0;
for (size_t i = 0; i < num_rows.size(); i++) {
uint32_t id = _segments_rowid_map.size();
_segment_to_id_map.emplace(std::pair<RowsetId, uint32_t> {src_rowset_id, i}, id);
_id_to_segment_map.emplace_back(src_rowset_id, i);
_segments_rowid_map.emplace_back(std::vector<std::pair<uint32_t, uint32_t>>(
num_rows[i], std::pair<uint32_t, uint32_t>(UINT32_MAX, UINT32_MAX)));
+ mem_used += sizeof(std::pair<uint32_t, uint32_t>) * _segments_rowid_map[i].capacity();
+ }
+ //NOTE: manually count _segments_rowid_map's memory here, because _segments_rowid_map could be used by indexCompaction.
+ // indexCompaction is a thridparty code, it's too complex to modify it.
+ // refer compact_column.
+ mem_used +=
+ sizeof(std::vector<std::pair<uint32_t, uint32_t>>) * _segments_rowid_map.capacity();
+ _seg_rowid_map_mem_used = mem_used;
+ auto* t_ctx = doris::thread_context(true);
+ if (t_ctx) {
Review Comment:
这里也不用保存，直接consume 就可以
##########
be/src/olap/rowid_conversion.h:
##########
@@ -115,6 +135,8 @@ class RowIdConversion {
// value indicates row id of destination segment.
// <UINT32_MAX, UINT32_MAX> indicates current row not exist.
std::vector<std::vector<std::pair<uint32_t, uint32_t>>> _segments_rowid_map;
+ size_t _seg_rowid_map_mem_used;
+ std::shared_ptr<MemTrackerLimiter> _row_id_thread_ctx_memtracker_ptr = nullptr;
Review Comment:
不用保存这个
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2370657303
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2372820307
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2375889121
<details>
<summary>TPC-DS: <b>Total hot run time: 192538 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 87e606d0bcccd09cbca56f91e480faf7b28bfd34, data reload: false
query1 960 381 377 377
query2 6356 2118 2020 2020
query3 8681 199 208 199
query4 34297 23493 23429 23429
query5 3376 464 477 464
query6 270 164 160 160
query7 4193 301 302 301
query8 277 222 221 221
query9 9593 2672 2670 2670
query10 458 277 272 272
query11 17956 15204 15241 15204
query12 156 100 99 99
query13 1533 430 410 410
query14 9638 7266 7559 7266
query15 254 166 180 166
query16 8031 473 491 473
query17 1689 605 618 605
query18 2174 337 324 324
query19 375 148 154 148
query20 120 111 125 111
query21 216 119 105 105
query22 4665 4452 4696 4452
query23 35309 34684 34507 34507
query24 11171 2851 2843 2843
query25 620 406 424 406
query26 1167 180 159 159
query27 2610 304 291 291
query28 7497 2440 2443 2440
query29 847 440 442 440
query30 262 156 151 151
query31 993 786 808 786
query32 93 54 55 54
query33 747 301 294 294
query34 907 509 494 494
query35 877 718 735 718
query36 1093 946 915 915
query37 146 80 84 80
query38 3973 3879 3888 3879
query39 1516 1401 1422 1401
query40 203 100 100 100
query41 51 49 49 49
query42 115 94 94 94
query43 527 488 486 486
query44 1246 822 795 795
query45 197 169 167 167
query46 1150 714 732 714
query47 1924 1838 1825 1825
query48 469 372 377 372
query49 902 403 394 394
query50 841 409 412 409
query51 7038 6946 7014 6946
query52 101 89 89 89
query53 254 178 203 178
query54 1105 460 478 460
query55 75 76 75 75
query56 283 267 251 251
query57 1236 1091 1116 1091
query58 235 229 241 229
query59 3142 2878 2970 2878
query60 288 266 276 266
query61 103 106 105 105
query62 841 654 651 651
query63 226 183 180 180
query64 4424 642 636 636
query65 3251 3192 3194 3192
query66 889 298 305 298
query67 16042 15664 15613 15613
query68 5065 555 572 555
query69 602 301 301 301
query70 1188 1179 1143 1143
query71 434 271 274 271
query72 7608 4093 4098 4093
query73 763 348 355 348
query74 10440 9002 9173 9002
query75 3796 2689 2698 2689
query76 3537 897 840 840
query77 519 312 299 299
query78 10575 9597 9597 9597
query79 2252 584 608 584
query80 2695 438 456 438
query81 580 238 235 235
query82 708 140 140 140
query83 297 133 138 133
query84 282 81 74 74
query85 1775 299 282 282
query86 471 314 312 312
query87 4390 4313 4340 4313
query88 3806 2446 2419 2419
query89 406 282 294 282
query90 2163 185 191 185
query91 171 163 142 142
query92 66 47 46 46
query93 2400 545 556 545
query94 1269 306 294 294
query95 367 253 261 253
query96 633 283 288 283
query97 3257 3100 3088 3088
query98 222 214 194 194
query99 1523 1287 1281 1281
Total cold run time: 306514 ms
Total hot run time: 192538 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on code in PR #41193:
URL: https://github.com/apache/doris/pull/41193#discussion_r1772569212
##########
be/src/olap/rowid_conversion.h:
##########
@@ -115,6 +121,7 @@ class RowIdConversion {
// value indicates row id of destination segment.
// <UINT32_MAX, UINT32_MAX> indicates current row not exist.
std::vector<std::vector<std::pair<uint32_t, uint32_t>>> _segments_rowid_map;
Review Comment:
because _segments_rowid_map is used by clucene, modify _segments_rowid_map's template means clucene's code should alse be modifed, it's too complex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2372938167
TeamCity be ut coverage result:
Function Coverage: 37.33% (9628/25791)
Line Coverage: 28.73% (79675/277311)
Region Coverage: 28.17% (41199/146264)
Branch Coverage: 24.79% (20985/84660)
Coverage Report: http://coverage.selectdb-in.cc/coverage/71fa09c491d5d5e5021ab2e57ea87a284a002de9_71fa09c491d5d5e5021ab2e57ea87a284a002de9/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2370062912
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2370142901
<details>
<summary>TPC-H: <b>Total hot run time: 41665 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 100d0a053351c850682d94e3ffa228f47257aae8, data reload: false
------ Round 1 ----------------------------------
q1 17571 7349 7235 7235
q2 2022 293 289 289
q3 12213 1064 1238 1064
q4 10550 762 787 762
q5 7771 3085 3081 3081
q6 239 155 154 154
q7 1016 612 650 612
q8 9455 2104 2030 2030
q9 6769 6376 6382 6376
q10 7010 2282 2275 2275
q11 432 249 254 249
q12 416 233 219 219
q13 17782 2973 2994 2973
q14 232 211 218 211
q15 586 532 508 508
q16 668 627 597 597
q17 983 873 873 873
q18 7285 6735 6750 6735
q19 1411 912 990 912
q20 576 292 296 292
q21 4017 3204 3356 3204
q22 1137 1022 1014 1014
Total cold run time: 110141 ms
Total hot run time: 41665 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7246 7265 7253 7253
q2 339 234 242 234
q3 3069 3010 2935 2935
q4 2076 1892 1785 1785
q5 5692 5562 5716 5562
q6 235 150 147 147
q7 2177 1813 1833 1813
q8 3291 3466 3465 3465
q9 8789 8805 8830 8805
q10 3482 3506 3467 3467
q11 600 482 480 480
q12 838 645 619 619
q13 9612 3196 3142 3142
q14 299 285 290 285
q15 584 550 529 529
q16 727 681 653 653
q17 1817 1620 1625 1620
q18 8161 7972 7866 7866
q19 1711 1530 1509 1509
q20 2117 1871 1919 1871
q21 5584 5485 5442 5442
q22 1186 1053 1055 1053
Total cold run time: 69632 ms
Total hot run time: 60535 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2370161381
<details>
<summary>ClickBench: <b>Total hot run time: 32.31 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 100d0a053351c850682d94e3ffa228f47257aae8, data reload: false
query1 0.04 0.05 0.04
query2 0.06 0.03 0.03
query3 0.24 0.07 0.07
query4 1.64 0.09 0.10
query5 0.50 0.50 0.53
query6 1.14 0.73 0.71
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.56 0.50 0.50
query10 0.54 0.55 0.54
query11 0.14 0.11 0.11
query12 0.14 0.11 0.11
query13 0.60 0.59 0.59
query14 2.99 2.93 3.02
query15 0.89 0.80 0.83
query16 0.36 0.36 0.39
query17 1.07 1.03 1.02
query18 0.20 0.20 0.19
query19 1.93 1.84 1.95
query20 0.01 0.01 0.01
query21 15.37 0.56 0.56
query22 2.94 2.01 1.77
query23 17.32 0.80 0.93
query24 2.72 0.80 2.09
query25 0.24 0.06 0.07
query26 0.51 0.14 0.13
query27 0.03 0.04 0.04
query28 10.20 1.09 1.07
query29 12.58 3.26 3.22
query30 0.25 0.05 0.06
query31 2.89 0.39 0.37
query32 3.28 0.48 0.47
query33 2.95 2.95 3.08
query34 17.03 4.43 4.37
query35 4.43 4.41 4.45
query36 0.65 0.48 0.46
query37 0.08 0.05 0.07
query38 0.04 0.03 0.03
query39 0.03 0.02 0.02
query40 0.16 0.13 0.13
query41 0.08 0.02 0.02
query42 0.04 0.03 0.02
query43 0.03 0.03 0.03
Total cold run time: 106.96 s
Total hot run time: 32.31 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2375877188
TeamCity be ut coverage result:
Function Coverage: 37.30% (9632/25822)
Line Coverage: 28.70% (79682/277619)
Region Coverage: 28.12% (41199/146492)
Branch Coverage: 24.76% (20981/84754)
Coverage Report: http://coverage.selectdb-in.cc/coverage/87e606d0bcccd09cbca56f91e480faf7b28bfd34_87e606d0bcccd09cbca56f91e480faf7b28bfd34/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2373736470
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2372893216
<details>
<summary>TPC-H: <b>Total hot run time: 41191 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 71fa09c491d5d5e5021ab2e57ea87a284a002de9, data reload: false
------ Round 1 ----------------------------------
q1 17957 7611 7365 7365
q2 2297 169 183 169
q3 11285 1242 1205 1205
q4 10586 726 728 726
q5 8282 2960 2917 2917
q6 241 155 158 155
q7 1001 638 613 613
q8 9851 2013 1924 1924
q9 6547 6443 6487 6443
q10 6993 2340 2354 2340
q11 450 253 253 253
q12 420 220 228 220
q13 17786 2976 2991 2976
q14 240 222 215 215
q15 575 525 521 521
q16 663 599 584 584
q17 996 555 534 534
q18 7308 6627 6761 6627
q19 1431 1033 1085 1033
q20 602 289 277 277
q21 4523 3237 3093 3093
q22 1090 1008 1001 1001
Total cold run time: 111124 ms
Total hot run time: 41191 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7263 7266 7289 7266
q2 328 235 228 228
q3 2912 2775 2858 2775
q4 1949 1714 1676 1676
q5 5459 5505 5516 5505
q6 233 143 146 143
q7 2148 1726 1714 1714
q8 3263 3439 3453 3439
q9 8575 8567 8570 8567
q10 3494 3459 3417 3417
q11 583 480 485 480
q12 762 570 584 570
q13 5256 3046 3022 3022
q14 307 254 282 254
q15 568 525 516 516
q16 670 638 647 638
q17 1797 1571 1603 1571
q18 7744 7533 7353 7353
q19 1680 1529 1455 1455
q20 2076 1811 1820 1811
q21 5503 5265 5259 5259
q22 1097 985 980 980
Total cold run time: 63667 ms
Total hot run time: 58639 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2373920032
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2375894125
<details>
<summary>ClickBench: <b>Total hot run time: 33.11 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 87e606d0bcccd09cbca56f91e480faf7b28bfd34, data reload: false
query1 0.05 0.04 0.04
query2 0.06 0.03 0.03
query3 0.23 0.06 0.06
query4 1.68 0.09 0.10
query5 0.51 0.50 0.52
query6 1.13 0.72 0.72
query7 0.02 0.02 0.02
query8 0.05 0.03 0.04
query9 0.56 0.51 0.50
query10 0.55 0.59 0.55
query11 0.13 0.11 0.11
query12 0.13 0.11 0.10
query13 0.61 0.59 0.60
query14 2.71 2.72 2.84
query15 0.91 0.82 0.83
query16 0.39 0.36 0.38
query17 1.03 0.97 1.04
query18 0.20 0.19 0.19
query19 1.94 1.88 2.04
query20 0.02 0.01 0.01
query21 15.35 0.60 0.57
query22 2.53 2.74 2.13
query23 17.56 0.83 0.83
query24 2.96 1.29 1.61
query25 0.18 0.12 0.06
query26 0.58 0.14 0.13
query27 0.04 0.04 0.04
query28 9.75 1.11 1.06
query29 12.53 3.20 3.17
query30 0.24 0.06 0.05
query31 2.89 0.37 0.38
query32 3.29 0.47 0.46
query33 2.97 2.98 3.03
query34 16.78 4.49 4.44
query35 4.48 4.53 4.47
query36 0.67 0.49 0.48
query37 0.08 0.05 0.06
query38 0.05 0.03 0.04
query39 0.03 0.02 0.02
query40 0.15 0.12 0.12
query41 0.07 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 106.16 s
Total hot run time: 33.11 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2376072432
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2374080104
<details>
<summary>TPC-H: <b>Total hot run time: 41354 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f229b65dd9ffcb607392fe50a80656f46a13aa52, data reload: false
------ Round 1 ----------------------------------
q1 18162 7613 7304 7304
q2 2545 164 160 160
q3 11112 1785 1192 1192
q4 10646 763 783 763
q5 8112 2965 2945 2945
q6 243 158 156 156
q7 979 642 625 625
q8 9453 1947 1943 1943
q9 6585 6413 6383 6383
q10 6946 2300 2338 2300
q11 442 254 246 246
q12 422 224 218 218
q13 17786 2975 2968 2968
q14 238 222 216 216
q15 567 516 513 513
q16 639 592 589 589
q17 1002 571 576 571
q18 7350 6716 6709 6709
q19 1410 1080 1051 1051
q20 599 288 278 278
q21 4120 3321 3240 3240
q22 1087 984 1012 984
Total cold run time: 110445 ms
Total hot run time: 41354 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7317 7221 7209 7209
q2 339 237 234 234
q3 2908 2763 2771 2763
q4 1977 1698 1735 1698
q5 5458 5491 5491 5491
q6 230 140 141 140
q7 2148 1773 1738 1738
q8 3299 3461 3468 3461
q9 8633 8668 8663 8663
q10 3519 3433 3455 3433
q11 577 474 490 474
q12 767 556 589 556
q13 5352 2987 2997 2987
q14 286 288 269 269
q15 564 520 508 508
q16 656 624 623 623
q17 1848 1583 1561 1561
q18 7895 7493 7362 7362
q19 1677 1458 1567 1458
q20 2033 1795 1837 1795
q21 5423 5291 5221 5221
q22 1096 1038 999 999
Total cold run time: 64002 ms
Total hot run time: 58643 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2374183422
TeamCity be ut coverage result:
Function Coverage: 37.30% (9631/25819)
Line Coverage: 28.70% (79657/277538)
Region Coverage: 28.13% (41195/146457)
Branch Coverage: 24.76% (20981/84742)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f229b65dd9ffcb607392fe50a80656f46a13aa52_f229b65dd9ffcb607392fe50a80656f46a13aa52/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2372850085
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2370063082
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2370690566
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #41193:
URL: https://github.com/apache/doris/pull/41193#discussion_r1774395214
##########
be/src/olap/rowid_conversion.h:
##########
@@ -22,6 +22,7 @@
#include "olap/olap_common.h"
#include "olap/utils.h"
+#include "runtime/thread_context.h"
namespace doris {
Review Comment:
thread1--> compaction ->switch
thread2 --> do compaction ---> init90
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2372871204
We dont need assume memtracker == nullptr OR thread changed during rowid conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2373901795
TeamCity be ut coverage result:
Function Coverage: 37.31% (9633/25816)
Line Coverage: 28.71% (79668/277528)
Region Coverage: 28.13% (41190/146450)
Branch Coverage: 24.76% (20979/84740)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2e18f21b8dadab8060f5f6ce17cd76f447927009_2e18f21b8dadab8060f5f6ce17cd76f447927009/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2372938011
<details>
<summary>ClickBench: <b>Total hot run time: 32.29 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 71fa09c491d5d5e5021ab2e57ea87a284a002de9, data reload: false
query1 0.05 0.04 0.04
query2 0.06 0.03 0.03
query3 0.23 0.06 0.06
query4 1.64 0.10 0.10
query5 0.52 0.52 0.52
query6 1.13 0.73 0.73
query7 0.02 0.01 0.01
query8 0.03 0.03 0.03
query9 0.56 0.50 0.50
query10 0.55 0.56 0.56
query11 0.14 0.11 0.11
query12 0.15 0.12 0.11
query13 0.61 0.63 0.60
query14 2.74 2.72 2.83
query15 0.90 0.83 0.83
query16 0.38 0.38 0.40
query17 1.05 1.05 1.06
query18 0.23 0.22 0.22
query19 1.96 1.90 1.96
query20 0.01 0.01 0.01
query21 15.36 0.60 0.63
query22 2.52 2.43 2.10
query23 17.39 0.86 0.75
query24 3.36 1.27 0.14
query25 0.19 0.07 0.05
query26 0.38 0.15 0.14
query27 0.04 0.04 0.06
query28 11.15 1.09 1.07
query29 12.57 3.22 3.22
query30 0.24 0.06 0.05
query31 2.88 0.39 0.39
query32 3.25 0.46 0.47
query33 2.98 3.02 3.07
query34 17.00 4.53 4.45
query35 4.53 4.54 4.53
query36 0.67 0.48 0.47
query37 0.09 0.06 0.06
query38 0.04 0.03 0.03
query39 0.04 0.02 0.02
query40 0.17 0.12 0.12
query41 0.07 0.02 0.02
query42 0.04 0.02 0.03
query43 0.03 0.03 0.03
Total cold run time: 107.95 s
Total hot run time: 32.29 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2373893404
<details>
<summary>TPC-H: <b>Total hot run time: 41233 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e18f21b8dadab8060f5f6ce17cd76f447927009, data reload: false
------ Round 1 ----------------------------------
q1 17566 7461 7300 7300
q2 2048 286 272 272
q3 12138 1095 1239 1095
q4 10568 782 734 734
q5 7762 2903 2856 2856
q6 243 151 148 148
q7 979 627 627 627
q8 9379 1989 2001 1989
q9 6604 6442 6418 6418
q10 7033 2291 2345 2291
q11 431 248 239 239
q12 432 230 222 222
q13 17781 2977 2999 2977
q14 242 211 212 211
q15 571 525 532 525
q16 654 584 582 582
q17 994 551 567 551
q18 7217 6801 6661 6661
q19 1397 1026 1069 1026
q20 589 293 295 293
q21 4178 3233 3242 3233
q22 1109 1008 983 983
Total cold run time: 109915 ms
Total hot run time: 41233 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7896 7283 7294 7283
q2 330 230 233 230
q3 3096 2965 2916 2916
q4 2112 1821 1867 1821
q5 5726 5681 5776 5681
q6 230 150 142 142
q7 2220 1839 1788 1788
q8 3393 3540 3540 3540
q9 8886 9029 8995 8995
q10 3527 3536 3582 3536
q11 585 489 475 475
q12 833 606 623 606
q13 8928 3217 3175 3175
q14 316 272 285 272
q15 595 533 528 528
q16 683 649 652 649
q17 1866 1642 1620 1620
q18 8354 7909 7902 7902
q19 1756 1702 1557 1557
q20 2136 1915 1882 1882
q21 5601 5493 5416 5416
q22 1111 1040 1023 1023
Total cold run time: 70180 ms
Total hot run time: 61037 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2374111477
<details>
<summary>TPC-DS: <b>Total hot run time: 191636 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f229b65dd9ffcb607392fe50a80656f46a13aa52, data reload: false
query1 982 373 384 373
query2 6503 2090 2020 2020
query3 6706 216 234 216
query4 34661 23570 23458 23458
query5 4502 479 474 474
query6 261 182 161 161
query7 4616 307 303 303
query8 288 237 230 230
query9 9781 2666 2653 2653
query10 466 312 291 291
query11 18104 15176 15307 15176
query12 152 101 101 101
query13 1627 445 423 423
query14 10690 7765 7263 7263
query15 320 174 174 174
query16 7863 474 471 471
query17 1800 595 564 564
query18 2029 314 323 314
query19 368 154 157 154
query20 117 104 111 104
query21 214 105 105 105
query22 4339 4071 4186 4071
query23 34908 34198 34132 34132
query24 11171 2933 2883 2883
query25 644 415 419 415
query26 1166 162 166 162
query27 2305 305 303 303
query28 7643 2448 2411 2411
query29 821 438 450 438
query30 323 156 155 155
query31 1038 807 829 807
query32 101 58 70 58
query33 796 327 315 315
query34 917 508 490 490
query35 880 756 733 733
query36 1113 976 949 949
query37 148 90 94 90
query38 4045 3978 3926 3926
query39 1474 1418 1413 1413
query40 207 101 102 101
query41 52 50 50 50
query42 118 97 94 94
query43 513 483 486 483
query44 1213 824 813 813
query45 197 170 170 170
query46 1170 726 741 726
query47 1893 1795 1862 1795
query48 493 373 385 373
query49 1163 433 408 408
query50 851 422 424 422
query51 6997 6938 7042 6938
query52 106 92 90 90
query53 267 192 194 192
query54 1237 484 492 484
query55 79 81 78 78
query56 297 289 285 285
query57 1211 1085 1092 1085
query58 249 252 256 252
query59 3256 3101 2919 2919
query60 326 286 292 286
query61 149 108 106 106
query62 855 673 667 667
query63 234 194 192 192
query64 4162 652 631 631
query65 3295 3175 3194 3175
query66 756 303 315 303
query67 15891 15562 15629 15562
query68 4859 585 556 556
query69 488 302 298 298
query70 1186 1099 1136 1099
query71 339 275 275 275
query72 6612 3965 4202 3965
query73 783 347 351 347
query74 9537 9142 8974 8974
query75 3400 2711 2645 2645
query76 2922 866 1035 866
query77 404 307 307 307
query78 10284 9521 9203 9203
query79 2288 605 619 605
query80 947 462 464 462
query81 598 244 249 244
query82 670 143 138 138
query83 251 145 164 145
query84 248 81 80 80
query85 1686 297 284 284
query86 491 300 301 300
query87 4478 4409 4313 4313
query88 4167 2408 2386 2386
query89 401 286 284 284
query90 1925 194 194 194
query91 192 147 146 146
query92 66 49 50 49
query93 1623 543 543 543
query94 858 314 289 289
query95 354 267 252 252
query96 614 285 290 285
query97 3253 3147 3138 3138
query98 226 199 194 194
query99 1562 1311 1328 1311
Total cold run time: 300831 ms
Total hot run time: 191636 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on code in PR #41193:
URL: https://github.com/apache/doris/pull/41193#discussion_r1772551660
##########
be/src/vec/common/custom_allocator.h:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "vec/common/allocator.h"
+#include "vec/common/allocator_fwd.h"
+
+template <class T, typename MemoryAllocator = Allocator<true>>
+class CustomStdAllocator;
+
+template <typename P>
+using MyVector = std::vector<P, CustomStdAllocator<P>>;
+
+template <typename K, typename T>
+using MyMap = std::map<K, T, std::less<K>, CustomStdAllocator<std::pair<const K, T>>>;
Review Comment:
maybe DorisMap sounds better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2375809847
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2375876665
<details>
<summary>TPC-H: <b>Total hot run time: 41015 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 87e606d0bcccd09cbca56f91e480faf7b28bfd34, data reload: false
------ Round 1 ----------------------------------
q1 17576 8104 7257 7257
q2 2019 288 291 288
q3 11960 1093 1225 1093
q4 10564 782 730 730
q5 7785 2886 2856 2856
q6 235 149 148 148
q7 998 619 598 598
q8 9370 1976 1942 1942
q9 6622 6497 6390 6390
q10 6947 2325 2335 2325
q11 435 253 255 253
q12 419 223 223 223
q13 17773 3006 2976 2976
q14 238 208 220 208
q15 584 530 520 520
q16 646 598 581 581
q17 995 569 552 552
q18 7178 6620 6793 6620
q19 1402 1064 1067 1064
q20 484 208 209 208
q21 4105 3298 3214 3214
q22 1093 969 1016 969
Total cold run time: 109428 ms
Total hot run time: 41015 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7235 7226 7241 7226
q2 327 230 232 230
q3 3066 2940 2972 2940
q4 2110 1925 1874 1874
q5 5724 5677 5710 5677
q6 231 153 148 148
q7 2202 1783 1777 1777
q8 3411 3532 3556 3532
q9 8897 9134 8972 8972
q10 3581 3541 3566 3541
q11 581 512 490 490
q12 873 627 661 627
q13 8553 3161 3198 3161
q14 310 273 290 273
q15 573 530 531 530
q16 702 645 646 645
q17 1884 1653 1596 1596
q18 8325 7670 7873 7670
q19 1726 1643 1535 1535
q20 2139 1877 1888 1877
q21 5541 5386 5413 5386
q22 1144 1043 1051 1043
Total cold run time: 69135 ms
Total hot run time: 60750 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2370155483
<details>
<summary>TPC-DS: <b>Total hot run time: 191736 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 100d0a053351c850682d94e3ffa228f47257aae8, data reload: false
query1 958 402 393 393
query2 6629 2072 1989 1989
query3 8683 195 207 195
query4 34055 23517 23558 23517
query5 3446 467 465 465
query6 279 166 171 166
query7 4197 299 290 290
query8 286 220 233 220
query9 9694 2705 2678 2678
query10 486 284 265 265
query11 18125 15138 15276 15138
query12 162 98 96 96
query13 1532 423 396 396
query14 8951 7632 7280 7280
query15 244 169 173 169
query16 7564 491 462 462
query17 1684 627 599 599
query18 2070 325 319 319
query19 392 158 154 154
query20 126 118 114 114
query21 218 108 106 106
query22 4865 4544 4398 4398
query23 35237 34322 34525 34322
query24 11345 3025 2970 2970
query25 629 406 414 406
query26 1211 159 161 159
query27 2355 283 294 283
query28 7927 2476 2475 2475
query29 820 438 432 432
query30 261 163 160 160
query31 1032 800 802 800
query32 95 53 54 53
query33 747 308 310 308
query34 906 500 493 493
query35 868 729 749 729
query36 1094 939 971 939
query37 158 92 88 88
query38 4000 3969 3952 3952
query39 1480 1403 1410 1403
query40 209 98 97 97
query41 50 49 49 49
query42 121 96 99 96
query43 532 481 496 481
query44 1296 835 822 822
query45 200 167 163 163
query46 1141 754 740 740
query47 1915 1845 1838 1838
query48 481 371 369 369
query49 886 408 407 407
query50 809 414 415 414
query51 7043 6741 6992 6741
query52 100 111 86 86
query53 251 181 178 178
query54 1156 468 457 457
query55 82 75 77 75
query56 294 257 244 244
query57 1228 1091 1092 1091
query58 229 229 226 226
query59 3334 2820 2824 2820
query60 293 275 263 263
query61 112 101 104 101
query62 836 663 675 663
query63 221 186 186 186
query64 4095 669 638 638
query65 3237 3189 3175 3175
query66 863 306 322 306
query67 15894 15587 15637 15587
query68 4728 606 588 588
query69 535 303 308 303
query70 1226 1125 1065 1065
query71 377 281 274 274
query72 7587 4067 4050 4050
query73 759 327 330 327
query74 10277 9041 8972 8972
query75 3471 2677 2665 2665
query76 3152 967 933 933
query77 619 303 286 286
query78 10011 9212 9153 9153
query79 1531 553 559 553
query80 1168 445 440 440
query81 565 240 242 240
query82 691 146 143 143
query83 302 134 133 133
query84 289 87 83 83
query85 1351 320 277 277
query86 393 306 296 296
query87 4535 4365 4373 4365
query88 3174 2323 2337 2323
query89 388 283 290 283
query90 1903 188 186 186
query91 172 146 143 143
query92 60 47 49 47
query93 1809 563 550 550
query94 923 298 290 290
query95 366 258 266 258
query96 611 279 275 275
query97 3271 3148 3063 3063
query98 212 214 192 192
query99 1611 1286 1291 1286
Total cold run time: 299786 ms
Total hot run time: 191736 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2370985577
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on code in PR #41193:
URL: https://github.com/apache/doris/pull/41193#discussion_r1772551014
##########
be/src/olap/rowid_conversion.h:
##########
@@ -33,17 +33,23 @@ namespace doris {
class RowIdConversion {
public:
RowIdConversion() = default;
- ~RowIdConversion() = default;
+ ~RowIdConversion() { CONSUME_THREAD_MEM_TRACKER(-_seg_rowid_map_mem_used); }
Review Comment:
RELEASE_THREAD_MEM_TRACKER
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2372918665
<details>
<summary>TPC-DS: <b>Total hot run time: 192376 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 71fa09c491d5d5e5021ab2e57ea87a284a002de9, data reload: false
query1 975 371 365 365
query2 6523 2117 2073 2073
query3 6701 213 224 213
query4 34875 23582 23675 23582
query5 4367 464 473 464
query6 268 173 168 168
query7 4631 324 310 310
query8 291 229 254 229
query9 9603 2669 2651 2651
query10 469 278 272 272
query11 18122 15230 15153 15153
query12 155 99 95 95
query13 1615 426 402 402
query14 9389 7778 7343 7343
query15 296 173 179 173
query16 7836 497 472 472
query17 1758 594 572 572
query18 1616 320 311 311
query19 357 153 155 153
query20 115 109 110 109
query21 220 104 107 104
query22 4673 4203 4072 4072
query23 34848 34374 34360 34360
query24 11178 2919 2855 2855
query25 677 401 422 401
query26 1372 162 165 162
query27 2729 295 306 295
query28 8039 2450 2447 2447
query29 866 457 442 442
query30 324 160 159 159
query31 1039 795 837 795
query32 103 60 61 60
query33 790 312 306 306
query34 952 509 521 509
query35 891 740 739 739
query36 1107 968 969 968
query37 169 88 95 88
query38 4009 3885 3880 3880
query39 1509 1442 1420 1420
query40 285 102 102 102
query41 54 49 51 49
query42 116 97 98 97
query43 548 496 503 496
query44 1234 829 807 807
query45 200 169 170 169
query46 1134 699 735 699
query47 1925 1808 1857 1808
query48 474 406 394 394
query49 1156 426 474 426
query50 833 414 423 414
query51 7147 7010 7035 7010
query52 101 89 90 89
query53 273 191 188 188
query54 1322 495 495 495
query55 81 79 81 79
query56 306 275 260 260
query57 1247 1128 1113 1113
query58 261 241 244 241
query59 3262 3020 2978 2978
query60 321 292 288 288
query61 127 129 124 124
query62 854 674 705 674
query63 229 195 185 185
query64 5311 649 622 622
query65 3324 3195 3200 3195
query66 1336 312 310 310
query67 15956 15782 15600 15600
query68 5116 573 572 572
query69 498 307 303 303
query70 1198 1183 1103 1103
query71 363 277 290 277
query72 7234 4059 4020 4020
query73 766 341 349 341
query74 10513 9201 8974 8974
query75 3721 2710 2665 2665
query76 3116 974 874 874
query77 639 302 301 301
query78 10185 9289 9281 9281
query79 2485 591 632 591
query80 1169 448 455 448
query81 561 244 253 244
query82 743 143 148 143
query83 319 139 135 135
query84 275 87 76 76
query85 1682 297 277 277
query86 495 304 313 304
query87 4502 4291 4342 4291
query88 4184 2433 2396 2396
query89 403 294 286 286
query90 2043 190 193 190
query91 181 149 144 144
query92 68 51 51 51
query93 1591 551 545 545
query94 1119 304 293 293
query95 361 259 261 259
query96 616 286 287 286
query97 3298 3139 3116 3116
query98 222 195 189 189
query99 1638 1338 1317 1317
Total cold run time: 306178 ms
Total hot run time: 192376 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2375800576
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo merged PR #41193:
URL: https://github.com/apache/doris/pull/41193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #41193:
URL: https://github.com/apache/doris/pull/41193#discussion_r1774371238
##########
be/src/vec/common/custom_allocator.h:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "vec/common/allocator.h"
+#include "vec/common/allocator_fwd.h"
+
+template <class T, typename MemoryAllocator = Allocator<true>>
+class CustomStdAllocator;
+
+template <typename P>
Review Comment:
template parameter is T
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Add customStdAllocator for vector/map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41193:
URL: https://github.com/apache/doris/pull/41193#issuecomment-2374126864
<details>
<summary>ClickBench: <b>Total hot run time: 32.35 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f229b65dd9ffcb607392fe50a80656f46a13aa52, data reload: false
query1 0.04 0.05 0.04
query2 0.06 0.02 0.03
query3 0.23 0.07 0.07
query4 1.65 0.10 0.10
query5 0.51 0.49 0.52
query6 1.14 0.72 0.72
query7 0.02 0.01 0.02
query8 0.03 0.03 0.03
query9 0.57 0.50 0.50
query10 0.55 0.56 0.55
query11 0.13 0.10 0.11
query12 0.13 0.11 0.11
query13 0.60 0.58 0.59
query14 2.82 2.72 2.83
query15 0.90 0.83 0.81
query16 0.38 0.40 0.38
query17 0.98 1.07 1.09
query18 0.23 0.20 0.21
query19 1.91 1.90 2.01
query20 0.01 0.01 0.01
query21 15.35 0.58 0.56
query22 2.74 1.81 1.50
query23 17.27 0.88 0.80
query24 3.16 1.05 1.28
query25 0.23 0.17 0.08
query26 0.51 0.14 0.15
query27 0.04 0.03 0.05
query28 10.34 1.10 1.06
query29 12.53 3.21 3.21
query30 0.24 0.06 0.06
query31 2.87 0.38 0.36
query32 3.30 0.47 0.46
query33 2.98 3.01 3.01
query34 17.01 4.50 4.43
query35 4.56 4.42 4.47
query36 0.68 0.48 0.47
query37 0.08 0.06 0.06
query38 0.05 0.03 0.03
query39 0.03 0.03 0.02
query40 0.15 0.12 0.13
query41 0.07 0.02 0.02
query42 0.04 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 107.16 s
Total hot run time: 32.35 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org