You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zclllyybb (via GitHub)" <gi...@apache.org> on 2025/10/13 15:30:06 UTC
[PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
zclllyybb opened a new pull request, #56921:
URL: https://github.com/apache/doris/pull/56921
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
1. forbid auto list partition + auto bucket
2. support auto range partition + auto bucket to calculate proper bucket for new partition. base on the assumption of incremental load of table.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3399964719
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3398051303
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3406928075
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3402134344
<details>
<summary>TPC-DS: <b>Total hot run time: 189592 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c99b080d075348334dd00a29d3f544240325247, data reload: false
query1 1100 431 406 406
query2 6578 1721 1689 1689
query3 6750 222 220 220
query4 26483 23385 23540 23385
query5 4571 669 488 488
query6 350 282 235 235
query7 4669 485 303 303
query8 310 264 253 253
query9 8720 2537 2575 2537
query10 514 331 287 287
query11 15733 15372 14812 14812
query12 185 128 109 109
query13 1672 542 432 432
query14 12669 9064 9200 9064
query15 212 192 173 173
query16 7680 663 516 516
query17 1636 806 618 618
query18 2083 457 345 345
query19 230 240 176 176
query20 136 143 134 134
query21 220 142 124 124
query22 4682 4769 4446 4446
query23 34854 33648 33932 33648
query24 8594 2562 2528 2528
query25 590 528 490 490
query26 1315 295 177 177
query27 3249 540 384 384
query28 4428 2180 2172 2172
query29 860 648 505 505
query30 330 249 205 205
query31 962 847 785 785
query32 91 69 73 69
query33 592 397 349 349
query34 826 887 539 539
query35 832 844 745 745
query36 973 1026 926 926
query37 126 113 129 113
query38 3471 3483 3460 3460
query39 1488 1435 1417 1417
query40 220 126 117 117
query41 59 58 61 58
query42 124 110 120 110
query43 467 496 461 461
query44 1325 837 811 811
query45 180 181 174 174
query46 838 991 624 624
query47 1762 1837 1740 1740
query48 395 418 313 313
query49 767 538 397 397
query50 632 698 405 405
query51 3830 3905 4005 3905
query52 114 112 100 100
query53 244 263 193 193
query54 590 595 535 535
query55 98 84 84 84
query56 327 335 309 309
query57 1233 1223 1130 1130
query58 284 282 288 282
query59 2555 2593 2546 2546
query60 359 344 310 310
query61 162 154 169 154
query62 801 757 669 669
query63 240 195 190 190
query64 4456 1176 838 838
query65 4055 3953 4017 3953
query66 1082 439 402 402
query67 15462 15225 15118 15118
query68 7920 930 599 599
query69 483 327 283 283
query70 1311 1238 1271 1238
query71 462 334 327 327
query72 5832 4867 4920 4867
query73 644 570 351 351
query74 8894 9196 8909 8909
query75 3567 3409 2765 2765
query76 3428 1157 720 720
query77 759 411 328 328
query78 9606 9731 8887 8887
query79 1820 851 592 592
query80 745 574 509 509
query81 536 270 228 228
query82 225 161 135 135
query83 286 275 252 252
query84 256 119 89 89
query85 874 474 429 429
query86 376 325 315 315
query87 3699 3725 3657 3657
query88 2820 2280 2221 2221
query89 395 321 298 298
query90 2048 224 220 220
query91 159 159 137 137
query92 91 74 70 70
query93 1765 1001 637 637
query94 681 443 306 306
query95 407 325 326 325
query96 487 580 283 283
query97 2918 2948 2871 2871
query98 259 217 214 214
query99 1339 1424 1338 1338
Total cold run time: 278517 ms
Total hot run time: 189592 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3402157010
<details>
<summary>ClickBench: <b>Total hot run time: 30.31 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c99b080d075348334dd00a29d3f544240325247, data reload: false
query1 0.05 0.05 0.05
query2 0.10 0.06 0.05
query3 0.26 0.08 0.08
query4 1.61 0.12 0.12
query5 0.28 0.26 0.25
query6 1.20 0.64 0.65
query7 0.03 0.02 0.02
query8 0.06 0.05 0.05
query9 0.64 0.53 0.52
query10 0.59 0.58 0.57
query11 0.17 0.11 0.11
query12 0.18 0.12 0.12
query13 0.64 0.62 0.63
query14 1.02 1.04 1.02
query15 0.86 0.84 0.87
query16 0.39 0.40 0.40
query17 1.10 1.09 1.07
query18 0.21 0.21 0.20
query19 1.98 1.84 1.83
query20 0.01 0.01 0.02
query21 15.43 0.91 0.62
query22 0.77 1.06 0.63
query23 15.22 1.38 0.60
query24 7.56 1.21 0.69
query25 0.49 0.30 0.09
query26 0.65 0.16 0.13
query27 0.06 0.06 0.06
query28 9.11 1.38 0.93
query29 12.63 3.99 3.28
query30 0.27 0.14 0.11
query31 2.82 0.61 0.39
query32 3.24 0.56 0.48
query33 3.10 3.06 3.16
query34 16.15 5.50 4.87
query35 4.91 4.95 4.89
query36 0.71 0.52 0.50
query37 0.11 0.08 0.08
query38 0.06 0.05 0.05
query39 0.04 0.03 0.03
query40 0.18 0.15 0.15
query41 0.08 0.03 0.03
query42 0.04 0.04 0.03
query43 0.04 0.04 0.04
Total cold run time: 105.05 s
Total hot run time: 30.31 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3408938536
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3402889374
# FE Regression Coverage Report
Increment line coverage ` 26.47% (9/34)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/56921_8c99b080d075348334dd00a29d3f544240325247_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/56921_8c99b080d075348334dd00a29d3f544240325247_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3398659377
<details>
<summary>ClickBench: <b>Total hot run time: 31.17 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dd32322438a461962cce70a110ac329e6db0bdf9, data reload: false
query1 0.05 0.05 0.06
query2 0.10 0.05 0.06
query3 0.24 0.08 0.08
query4 1.61 0.12 0.12
query5 0.27 0.27 0.26
query6 1.19 0.66 0.64
query7 0.03 0.03 0.03
query8 0.06 0.04 0.04
query9 0.62 0.52 0.52
query10 0.61 0.58 0.58
query11 0.16 0.11 0.12
query12 0.15 0.13 0.12
query13 0.64 0.61 0.61
query14 1.04 1.04 1.02
query15 0.86 0.87 0.87
query16 0.39 0.40 0.39
query17 1.03 1.08 1.06
query18 0.22 0.20 0.20
query19 1.95 1.80 1.82
query20 0.02 0.01 0.02
query21 15.44 0.93 0.59
query22 0.77 1.23 0.81
query23 14.75 1.38 0.62
query24 6.85 1.35 1.63
query25 0.50 0.26 0.09
query26 0.61 0.15 0.14
query27 0.06 0.06 0.06
query28 10.31 1.35 0.94
query29 12.55 3.91 3.21
query30 0.28 0.13 0.12
query31 2.83 0.56 0.38
query32 3.25 0.56 0.47
query33 3.09 3.21 3.12
query34 16.13 5.44 4.92
query35 4.94 4.89 4.98
query36 0.71 0.52 0.51
query37 0.10 0.08 0.07
query38 0.07 0.05 0.05
query39 0.04 0.04 0.04
query40 0.18 0.14 0.15
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 104.87 s
Total hot run time: 31.17 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3407303802
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #56921:
URL: https://github.com/apache/doris/pull/56921#discussion_r2426707529
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionExprUtil.java:
##########
@@ -187,7 +193,43 @@ public static Map<String, AddPartitionClause> getAddPartitionClauseFromPartition
}
Map<String, String> partitionProperties = Maps.newHashMap();
- DistributionDesc distributionDesc = olapTable.getDefaultDistributionInfo().toDistributionDesc();
+ DistributionInfo defaultDistributionInfo = olapTable.getDefaultDistributionInfo();
+ DistributionDesc distributionDesc = defaultDistributionInfo.toDistributionDesc();
+ if (olapTable.isAutoBucket() && partitionType == PartitionType.RANGE) {
+ List<Partition> partitions = DynamicPartitionScheduler.getHistoricalPartitions(olapTable, partitionName);
+ List<Long> visibleVersions;
+ try {
+ visibleVersions = Partition.getVisibleVersions(partitions);
+ } catch (RpcException e) {
+ LOG.warn("auto bucket get visible version fail, table: [{}-{}], partition: {}",
+ olapTable.getName(), olapTable.getId(), partitionName, e);
Review Comment:
The error message includes a placeholder for exception details but doesn't include the exception in the log parameters. Add the exception 'e' as the last parameter to properly log the stack trace.
##########
fe/fe-core/src/test/java/org/apache/doris/analysis/PartitionExprUtilTest.java:
##########
@@ -0,0 +1,219 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.FeConstants;
+import org.apache.doris.service.ExecuteEnv;
+import org.apache.doris.service.FrontendServiceImpl;
+import org.apache.doris.thrift.TCreatePartitionRequest;
+import org.apache.doris.thrift.TCreatePartitionResult;
+import org.apache.doris.thrift.TNullableStringLiteral;
+import org.apache.doris.thrift.TStatusCode;
+import org.apache.doris.utframe.TestWithFeService;
+
+import mockit.Expectations;
+import mockit.Mocked;
+import org.apache.doris.clone.DynamicPartitionScheduler;
+import org.apache.doris.common.Pair;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class PartitionExprUtilTest extends TestWithFeService {
+ private static final Logger LOG = LogManager.getLogger(PartitionExprUtilTest.class);
+
+ @Mocked
+ ExecuteEnv exeEnv;
+ @Override
+ protected void runBeforeAll() throws Exception {
+ FeConstants.runningUnitTest = true;
+ FeConstants.default_scheduler_interval_millisecond = 100;
+ Config.dynamic_partition_enable = true;
+ Config.dynamic_partition_check_interval_seconds = 1;
+ createDatabase("test");
+ }
+
+ @Test
+ public void testAutoBucketLargeDataCalculatesBuckets() throws Exception {
+ String createOlapTblStmt = "CREATE TABLE test.auto_bucket_calc_large (\n"
+ + " event_day DATETIME NOT NULL,\n"
+ + " site_id INT,\n"
+ + " v INT\n"
+ + ")\n"
+ + "DUPLICATE KEY(event_day, site_id)\n"
+ + "AUTO PARTITION BY range (date_trunc(event_day,'day')) (\n"
+ + "\n"
+ + ")\n"
+ + "DISTRIBUTED BY HASH(event_day) BUCKETS AUTO\n"
+ + "PROPERTIES(\"replication_num\"=\"1\");";
+
+ createTable(createOlapTblStmt);
+ Database db = Env.getCurrentInternalCatalog().getDbOrAnalysisException("test");
+ OlapTable table = (OlapTable) db.getTableOrAnalysisException("auto_bucket_calc_large");
+
+ FrontendServiceImpl impl = new FrontendServiceImpl(exeEnv);
+
+ // Create 3 historical partitions: 2023-08-01, 02, 03
+ String[] days = {"2023-08-01 00:00:00", "2023-08-02 00:00:00", "2023-08-03 00:00:00"};
+ String[] partNames = {"p20230801000000", "p20230802000000", "p20230803000000"};
+ for (String day : days) {
+ List<List<TNullableStringLiteral>> partitionValues = new ArrayList<>();
+ List<TNullableStringLiteral> values = new ArrayList<>();
+ TNullableStringLiteral start = new TNullableStringLiteral();
+ start.setValue(day);
+ values.add(start);
+ partitionValues.add(values);
+
+ TCreatePartitionRequest request = new TCreatePartitionRequest();
+ request.setDbId(db.getId());
+ request.setTableId(table.getId());
+ request.setPartitionValues(partitionValues);
+ TCreatePartitionResult result = impl.createPartition(request);
+ Assertions.assertEquals(TStatusCode.OK, result.getStatus().getStatusCode());
+ }
+
+ // Mark them as having data (visibleVersion >= 2)
+ table.writeLockOrDdlException();
+ try {
+ for (String pn : partNames) {
+ Partition p = table.getPartition(pn);
+ Assertions.assertNotNull(p);
+ p.setVisibleVersionAndTime(2L, System.currentTimeMillis());
+ }
+ } finally {
+ table.writeUnlock();
+ }
+
+ // Mock large compressed data sizes for each historical partition
+ long GB = 1024L * 1024L * 1024L;
+ Partition p1 = table.getPartition(partNames[0]);
+ Partition p2 = table.getPartition(partNames[1]);
+ Partition p3 = table.getPartition(partNames[2]);
+ new Expectations(p1, p2, p3) {{
+ p1.getDataSizeExcludeEmptyReplica(true); result = 200L * GB; minTimes = 0; // ~1 TB uncompressed
+ p2.getDataSizeExcludeEmptyReplica(true); result = 300L * GB; minTimes = 0; // ~1.5 TB uncompressed
+ p3.getDataSizeExcludeEmptyReplica(true); result = 400L * GB; minTimes = 0; // ~2 TB uncompressed
Review Comment:
The comments indicate uncompressed sizes (~1 TB, ~1.5 TB, ~2 TB) but the mocked values are for compressed sizes (200GB, 300GB, 400GB). The comments should reflect the actual mocked compressed sizes or clarify the compression ratio assumption.
```suggestion
p1.getDataSizeExcludeEmptyReplica(true); result = 200L * GB; minTimes = 0; // 200 GB compressed
p2.getDataSizeExcludeEmptyReplica(true); result = 300L * GB; minTimes = 0; // 300 GB compressed
p3.getDataSizeExcludeEmptyReplica(true); result = 400L * GB; minTimes = 0; // 400 GB compressed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3398483560
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3399964783
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3398642596
<details>
<summary>TPC-DS: <b>Total hot run time: 190630 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd32322438a461962cce70a110ac329e6db0bdf9, data reload: false
query1 1073 425 395 395
query2 6568 1699 1720 1699
query3 6759 221 214 214
query4 26221 23579 23210 23210
query5 5173 619 475 475
query6 371 238 238 238
query7 4667 485 300 300
query8 298 253 248 248
query9 8717 2552 2571 2552
query10 503 335 313 313
query11 15736 15070 14989 14989
query12 185 123 122 122
query13 1671 547 424 424
query14 11821 9225 9246 9225
query15 202 189 182 182
query16 7712 699 523 523
query17 1354 730 615 615
query18 2053 466 355 355
query19 328 252 200 200
query20 137 136 139 136
query21 220 136 116 116
query22 4813 4952 4953 4952
query23 35478 34163 33846 33846
query24 8446 2537 2583 2537
query25 624 551 510 510
query26 1367 288 169 169
query27 2991 533 375 375
query28 4637 2215 2222 2215
query29 852 681 578 578
query30 316 239 203 203
query31 943 861 761 761
query32 91 84 74 74
query33 657 396 364 364
query34 896 882 562 562
query35 870 877 754 754
query36 982 1020 918 918
query37 121 108 83 83
query38 3511 3531 3469 3469
query39 1456 1402 1415 1402
query40 225 127 114 114
query41 61 60 55 55
query42 119 105 119 105
query43 491 499 495 495
query44 1314 831 819 819
query45 186 179 171 171
query46 844 1014 637 637
query47 1775 1822 1764 1764
query48 388 442 308 308
query49 759 489 415 415
query50 644 684 406 406
query51 3865 3901 3890 3890
query52 106 105 97 97
query53 233 268 192 192
query54 595 582 530 530
query55 86 81 82 81
query56 306 302 310 302
query57 1181 1195 1157 1157
query58 287 276 277 276
query59 2508 2668 2560 2560
query60 344 345 330 330
query61 152 173 155 155
query62 810 744 686 686
query63 230 198 194 194
query64 4429 1143 831 831
query65 4046 3948 3995 3948
query66 1066 430 336 336
query67 15705 15372 15086 15086
query68 9718 925 582 582
query69 494 333 278 278
query70 1349 1390 1250 1250
query71 508 346 317 317
query72 5545 4906 4805 4805
query73 709 573 363 363
query74 9162 8831 9083 8831
query75 4480 3370 2790 2790
query76 4941 1162 742 742
query77 1040 403 337 337
query78 9648 9619 9055 9055
query79 5885 805 571 571
query80 738 561 509 509
query81 487 275 230 230
query82 428 159 138 138
query83 297 273 251 251
query84 303 121 91 91
query85 862 529 421 421
query86 334 339 289 289
query87 3776 3821 3632 3632
query88 2851 2297 2223 2223
query89 446 333 285 285
query90 2145 209 223 209
query91 154 163 132 132
query92 83 71 61 61
query93 3636 966 638 638
query94 688 413 343 343
query95 403 323 310 310
query96 488 636 278 278
query97 2931 3020 2872 2872
query98 250 233 227 227
query99 1415 1425 1327 1327
Total cold run time: 289769 ms
Total hot run time: 190630 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3407579021
# FE UT Coverage Report
Increment line coverage ` 82.02% (73/89)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/56921_6539e51a374a2461eb77746babb9968182435f62/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/56921_6539e51a374a2461eb77746babb9968182435f62/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb merged PR #56921:
URL: https://github.com/apache/doris/pull/56921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3399042895
# FE Regression Coverage Report
Increment line coverage ` 26.47% (9/34)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/56921_dd32322438a461962cce70a110ac329e6db0bdf9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/56921_dd32322438a461962cce70a110ac329e6db0bdf9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3401946201
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #56921:
URL: https://github.com/apache/doris/pull/56921#discussion_r2431260873
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionExprUtil.java:
##########
@@ -187,7 +193,44 @@ public static Map<String, AddPartitionClause> getAddPartitionClauseFromPartition
}
Map<String, String> partitionProperties = Maps.newHashMap();
- DistributionDesc distributionDesc = olapTable.getDefaultDistributionInfo().toDistributionDesc();
+ DistributionInfo defaultDistributionInfo = olapTable.getDefaultDistributionInfo();
+ DistributionDesc distributionDesc = defaultDistributionInfo.toDistributionDesc();
+ if (olapTable.isAutoBucket() && partitionType == PartitionType.RANGE) {
+ List<Partition> partitions = DynamicPartitionScheduler.getHistoricalPartitions(olapTable,
Review Comment:
这里看 能不能和 auto bucket 计算的地方，抽成一个函数，感觉有冗余，不好维护
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3407439486
<details>
<summary>ClickBench: <b>Total hot run time: 30.54 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6539e51a374a2461eb77746babb9968182435f62, data reload: false
query1 0.06 0.05 0.05
query2 0.10 0.05 0.05
query3 0.25 0.08 0.08
query4 1.60 0.12 0.11
query5 0.27 0.26 0.25
query6 1.20 0.66 0.64
query7 0.04 0.03 0.03
query8 0.05 0.04 0.05
query9 0.64 0.52 0.51
query10 0.58 0.60 0.58
query11 0.16 0.11 0.12
query12 0.16 0.12 0.12
query13 0.63 0.62 0.62
query14 1.03 1.03 1.02
query15 0.87 0.85 0.86
query16 0.39 0.40 0.40
query17 1.09 1.10 1.08
query18 0.21 0.19 0.20
query19 1.94 1.81 1.86
query20 0.02 0.01 0.02
query21 15.43 0.92 0.57
query22 0.75 1.12 0.62
query23 15.08 1.41 0.65
query24 7.14 1.02 1.20
query25 0.41 0.21 0.08
query26 0.60 0.16 0.15
query27 0.06 0.06 0.06
query28 9.83 1.35 0.93
query29 12.54 4.02 3.24
query30 0.29 0.13 0.14
query31 2.83 0.62 0.40
query32 3.24 0.55 0.48
query33 3.18 3.25 3.10
query34 16.16 5.46 4.83
query35 4.88 4.96 4.88
query36 0.70 0.51 0.50
query37 0.13 0.08 0.07
query38 0.07 0.05 0.04
query39 0.04 0.02 0.03
query40 0.18 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.02
query43 0.04 0.03 0.03
Total cold run time: 105 s
Total hot run time: 30.54 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3397999010
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3398341944
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3407420348
<details>
<summary>TPC-DS: <b>Total hot run time: 190478 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6539e51a374a2461eb77746babb9968182435f62, data reload: false
query1 1081 434 411 411
query2 6575 1740 1674 1674
query3 6752 229 221 221
query4 26571 24156 23548 23548
query5 5695 646 480 480
query6 336 239 218 218
query7 4648 502 290 290
query8 330 271 254 254
query9 8722 2571 2604 2571
query10 557 342 281 281
query11 15570 15007 14841 14841
query12 193 112 112 112
query13 1675 573 435 435
query14 11190 9209 9258 9209
query15 214 201 178 178
query16 7666 686 539 539
query17 1355 785 627 627
query18 2096 493 369 369
query19 242 229 213 213
query20 137 142 140 140
query21 241 147 131 131
query22 4624 4902 4537 4537
query23 34728 33960 33668 33668
query24 9235 2455 2522 2455
query25 620 532 469 469
query26 1270 286 164 164
query27 2775 537 376 376
query28 4926 2242 2192 2192
query29 837 671 501 501
query30 301 243 212 212
query31 914 869 773 773
query32 91 69 70 69
query33 616 377 371 371
query34 819 890 558 558
query35 870 877 776 776
query36 1072 1057 983 983
query37 131 112 90 90
query38 3605 3667 3528 3528
query39 1552 1486 1491 1486
query40 227 133 116 116
query41 64 57 62 57
query42 121 112 108 108
query43 509 495 467 467
query44 1348 866 827 827
query45 186 180 177 177
query46 862 999 629 629
query47 1737 1817 1746 1746
query48 394 420 311 311
query49 773 510 422 422
query50 672 682 414 414
query51 3917 3824 3886 3824
query52 115 107 106 106
query53 239 267 196 196
query54 610 591 526 526
query55 82 91 83 83
query56 309 321 320 320
query57 1194 1203 1145 1145
query58 283 274 269 269
query59 2610 2698 2569 2569
query60 347 342 318 318
query61 153 151 157 151
query62 788 727 674 674
query63 234 197 190 190
query64 4480 1170 826 826
query65 4032 3940 3992 3940
query66 1039 431 315 315
query67 15700 15228 15183 15183
query68 7918 961 583 583
query69 500 319 286 286
query70 1366 1321 1270 1270
query71 498 344 312 312
query72 5713 5007 5023 5007
query73 576 714 359 359
query74 8829 9006 9072 9006
query75 4616 3322 2819 2819
query76 3384 1163 729 729
query77 940 405 322 322
query78 9768 9655 8870 8870
query79 2545 832 591 591
query80 712 547 490 490
query81 517 268 228 228
query82 500 159 136 136
query83 296 257 282 257
query84 300 117 100 100
query85 871 455 412 412
query86 355 320 296 296
query87 3730 3738 3596 3596
query88 3886 2209 2208 2208
query89 387 349 288 288
query90 2025 221 206 206
query91 173 162 132 132
query92 88 69 64 64
query93 1878 986 642 642
query94 683 445 339 339
query95 402 324 315 315
query96 502 580 285 285
query97 2935 2960 3010 2960
query98 245 215 216 215
query99 1464 1411 1296 1296
Total cold run time: 282459 ms
Total hot run time: 190478 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](auto partition) Auto partition could use auto bucket [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56921:
URL: https://github.com/apache/doris/pull/56921#issuecomment-3402236088
# FE UT Coverage Report
Increment line coverage ` 67.65% (23/34)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/56921_8c99b080d075348334dd00a29d3f544240325247/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/56921_8c99b080d075348334dd00a29d3f544240325247/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org