You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mongo360 (via GitHub)" <gi...@apache.org> on 2024/07/11 08:09:08 UTC
[PR] fix partition storage policy info miss when restart fe [doris]
mongo360 opened a new pull request, #37661:
URL: https://github.com/apache/doris/pull/37661
## Proposed changes
Issue Number: close #xxx
Problem:
partition storage policy info lost when cluster restart (FE restart).
Example:
1. Create table with storage policy set on some partitions
```
CREATE TABLE `ad_effects` (
`pin_id` bigint(20) NOT NULL COMMENT '广告主pin_id',
`date_time` datetime NOT NULL COMMENT '点击时间',
`ad_plan_id` bigint(20) NOT NULL COMMENT '广告计划id',
`ad_group_id` bigint(20) NOT NULL COMMENT '广告单元id',
`loc` int(11) NOT NULL COMMENT '站内外标识 1表示站内, 2表示站外（中间页包含在站内）',
`impressions` bigint(20) SUM NULL DEFAULT "0" COMMENT '用户总展现',
`clicks` bigint(20) SUM NULL DEFAULT "0" COMMENT '用户总点击',
`cost` bigint(20) SUM NULL DEFAULT "0" COMMENT '广告花费'
) ENGINE=OLAP
AGGREGATE KEY(`pin_id`, `date_time`, `ad_plan_id`, `ad_group_id`, `loc`)
COMMENT 'OLAP'
PARTITION BY RANGE(`date_time`)
(PARTITION p202401 VALUES [('2024-01-01 00:00:00'), ('2024-02-01 00:00:00'))PROPERTIES ("STORAGE POLICY" = "chubaofs_ssd_storage_policy"),
PARTITION p202402 VALUES [('2024-02-01 00:00:00'), ('2024-03-01 00:00:00')),
PARTITION p202403 VALUES [('2024-03-01 00:00:00'), ('2024-04-01 00:00:00')))
DISTRIBUTED BY HASH(`pin_id`) BUCKETS 16
PROPERTIES (
"replication_allocation" = "tag.location.default: 3",
"is_being_synced" = "false",
"dynamic_partition.enable" = "false",
"dynamic_partition.time_unit" = "MONTH",
"dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.start" = "-2147483648",
"dynamic_partition.end" = "5",
"dynamic_partition.prefix" = "p",
"dynamic_partition.replication_allocation" = "tag.location.default: 3",
"dynamic_partition.buckets" = "32",
"dynamic_partition.create_history_partition" = "false",
"dynamic_partition.history_partition_num" = "-1",
"dynamic_partition.hot_partition_num" = "0",
"dynamic_partition.reserved_history_periods" = "NULL",
"dynamic_partition.storage_policy" = "",
"dynamic_partition.start_day_of_month" = "1",
"storage_format" = "V2",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false"
);
```
2. Restart cluster (restart FEs)
3. show create table ad_effects
```
CREATE TABLE `ad_effects` (
`pin_id` bigint(20) NOT NULL COMMENT '广告主pin_id',
`date_time` datetime NOT NULL COMMENT '点击时间',
`ad_plan_id` bigint(20) NOT NULL COMMENT '广告计划id',
`ad_group_id` bigint(20) NOT NULL COMMENT '广告单元id',
`loc` int(11) NOT NULL COMMENT '站内外标识 1表示站内, 2表示站外（中间页包含在站内）',
`impressions` bigint(20) SUM NULL DEFAULT "0" COMMENT '用户总展现',
`clicks` bigint(20) SUM NULL DEFAULT "0" COMMENT '用户总点击',
`cost` bigint(20) SUM NULL DEFAULT "0" COMMENT '广告花费'
) ENGINE=OLAP
AGGREGATE KEY(`pin_id`, `date_time`, `ad_plan_id`, `ad_group_id`, `loc`)
COMMENT 'OLAP'
PARTITION BY RANGE(`date_time`)
(PARTITION p202401 VALUES [('2024-01-01 00:00:00'), ('2024-02-01 00:00:00')),
PARTITION p202402 VALUES [('2024-02-01 00:00:00'), ('2024-03-01 00:00:00')),
PARTITION p202403 VALUES [('2024-03-01 00:00:00'), ('2024-04-01 00:00:00')))
DISTRIBUTED BY HASH(`pin_id`) BUCKETS 16
PROPERTIES (
"replication_allocation" = "tag.location.default: 3",
"is_being_synced" = "false",
"dynamic_partition.enable" = "false",
"dynamic_partition.time_unit" = "MONTH",
"dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.start" = "-2147483648",
"dynamic_partition.end" = "5",
"dynamic_partition.prefix" = "p",
"dynamic_partition.replication_allocation" = "tag.location.default: 3",
"dynamic_partition.buckets" = "32",
"dynamic_partition.create_history_partition" = "false",
"dynamic_partition.history_partition_num" = "-1",
"dynamic_partition.hot_partition_num" = "0",
"dynamic_partition.reserved_history_periods" = "NULL",
"dynamic_partition.storage_policy" = "",
"dynamic_partition.start_day_of_month" = "1",
"storage_format" = "V2",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false"
);
```
we find STORAGE POLICY of partition p202401 is missing.
Reason:
idToStoragePolicy in PartitionInfo which saved the storage policy info of the partitions was not serialize to local mate disk。
Solved:
add storage policy info where serialize PartitionInfo in function write and read storage policy info where init PartitionInfo with readFields when FE start.
add FeMetaVersion to compatible history version。
related to https://github.com/apache/doris/pull/32852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](cooldown) partition storage policy info lost when FE restart [doris]
Posted by "GoGoWen (via GitHub)" <gi...@apache.org>.
GoGoWen commented on PR #37661:
URL: https://github.com/apache/doris/pull/37661#issuecomment-2222317722
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](cooldown) partition storage policy info lost when FE restart [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37661:
URL: https://github.com/apache/doris/pull/37661#issuecomment-2222447464
<details>
<summary>TPC-H: <b>Total hot run time: 40022 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 12214aca21fbe510eaae1ed46a54ecafd5df4413, data reload: false
------ Round 1 ----------------------------------
q1 18306 4736 4333 4333
q2 2028 208 186 186
q3 10525 1163 1085 1085
q4 10246 826 778 778
q5 7544 2706 2722 2706
q6 222 138 139 138
q7 976 611 603 603
q8 9218 2105 2088 2088
q9 8794 6570 6597 6570
q10 8730 3741 3777 3741
q11 472 238 246 238
q12 392 230 237 230
q13 17755 3019 2963 2963
q14 274 225 234 225
q15 527 485 485 485
q16 498 374 379 374
q17 957 671 708 671
q18 8106 7400 7358 7358
q19 3814 1532 1477 1477
q20 667 321 331 321
q21 5032 3118 3231 3118
q22 392 342 334 334
Total cold run time: 115475 ms
Total hot run time: 40022 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4383 4268 4226 4226
q2 370 280 268 268
q3 3021 2746 2919 2746
q4 1979 1731 1667 1667
q5 5650 5575 5522 5522
q6 228 136 147 136
q7 2163 1856 1892 1856
q8 3368 3442 3420 3420
q9 8730 8745 8852 8745
q10 4121 3945 3847 3847
q11 594 503 495 495
q12 808 644 648 644
q13 17236 3200 3195 3195
q14 306 292 309 292
q15 562 526 483 483
q16 488 425 434 425
q17 1824 1527 1501 1501
q18 8089 7981 7907 7907
q19 1810 1617 1632 1617
q20 2169 1881 1868 1868
q21 8996 5004 4689 4689
q22 641 558 562 558
Total cold run time: 77536 ms
Total hot run time: 56107 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](cooldown) partition storage policy info lost when FE restart [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37661:
URL: https://github.com/apache/doris/pull/37661#issuecomment-2222485358
<details>
<summary>ClickBench: <b>Total hot run time: 30.68 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 12214aca21fbe510eaae1ed46a54ecafd5df4413, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.22 0.05 0.05
query4 1.68 0.10 0.10
query5 0.50 0.49 0.48
query6 1.13 0.73 0.73
query7 0.01 0.01 0.01
query8 0.05 0.04 0.04
query9 0.56 0.49 0.50
query10 0.53 0.54 0.54
query11 0.15 0.11 0.11
query12 0.14 0.12 0.12
query13 0.60 0.59 0.59
query14 0.76 0.77 0.77
query15 0.85 0.81 0.80
query16 0.36 0.35 0.37
query17 1.01 1.05 1.03
query18 0.23 0.22 0.22
query19 1.79 1.68 1.85
query20 0.02 0.01 0.01
query21 15.39 0.74 0.64
query22 4.00 7.14 2.02
query23 18.26 1.40 1.24
query24 2.06 0.23 0.22
query25 0.14 0.08 0.09
query26 0.30 0.21 0.21
query27 0.46 0.24 0.23
query28 13.24 1.01 1.00
query29 12.63 3.28 3.27
query30 0.25 0.06 0.06
query31 2.87 0.39 0.40
query32 3.26 0.48 0.47
query33 2.84 2.93 2.98
query34 16.97 4.36 4.30
query35 4.40 4.38 4.44
query36 0.66 0.47 0.47
query37 0.19 0.16 0.16
query38 0.15 0.16 0.14
query39 0.04 0.03 0.04
query40 0.16 0.13 0.13
query41 0.10 0.05 0.05
query42 0.06 0.04 0.05
query43 0.05 0.04 0.03
Total cold run time: 109.19 s
Total hot run time: 30.68 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](cooldown) partition storage policy info lost when FE restart [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37661:
URL: https://github.com/apache/doris/pull/37661#issuecomment-2578926002
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](cooldown) partition storage policy info lost when FE restart [doris]
Posted by "mongo360 (via GitHub)" <gi...@apache.org>.
mongo360 commented on PR #37661:
URL: https://github.com/apache/doris/pull/37661#issuecomment-2224278443
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](cooldown) partition storage policy info lost when FE restart [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #37661: [Fix](cooldown) partition storage policy info lost when FE restart
URL: https://github.com/apache/doris/pull/37661
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] fix partition storage policy info miss when restart fe [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37661:
URL: https://github.com/apache/doris/pull/37661#issuecomment-2222301835
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](cooldown) partition storage policy info lost when FE restart [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37661:
URL: https://github.com/apache/doris/pull/37661#issuecomment-2222472770
<details>
<summary>TPC-DS: <b>Total hot run time: 175975 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 12214aca21fbe510eaae1ed46a54ecafd5df4413, data reload: false
query1 911 371 370 370
query2 6438 2384 2363 2363
query3 6630 216 222 216
query4 23621 17465 17329 17329
query5 3666 495 481 481
query6 282 192 164 164
query7 4578 303 293 293
query8 327 292 308 292
query9 8506 2469 2465 2465
query10 440 287 286 286
query11 10531 10145 10110 10110
query12 118 91 85 85
query13 1661 384 372 372
query14 9781 8181 7676 7676
query15 263 183 186 183
query16 7763 328 331 328
query17 1777 563 548 548
query18 1921 289 275 275
query19 185 147 158 147
query20 95 79 84 79
query21 197 127 126 126
query22 4266 4034 4082 4034
query23 34019 33631 33642 33631
query24 10578 2823 2897 2823
query25 590 389 389 389
query26 733 165 156 156
query27 2215 279 292 279
query28 6203 2178 2157 2157
query29 904 655 653 653
query30 250 150 150 150
query31 958 786 764 764
query32 100 57 53 53
query33 652 301 311 301
query34 887 510 517 510
query35 709 573 593 573
query36 1141 997 1007 997
query37 149 82 90 82
query38 2884 2860 2800 2800
query39 914 841 868 841
query40 226 120 122 120
query41 52 51 50 50
query42 117 100 102 100
query43 594 557 557 557
query44 1060 726 731 726
query45 193 165 169 165
query46 1094 744 749 744
query47 1859 1749 1766 1749
query48 378 291 317 291
query49 853 408 423 408
query50 778 402 405 402
query51 6910 6870 6753 6753
query52 106 93 96 93
query53 371 296 296 296
query54 887 455 454 454
query55 77 77 75 75
query56 286 270 270 270
query57 1163 1079 1059 1059
query58 244 265 263 263
query59 3598 3150 3048 3048
query60 309 294 298 294
query61 134 111 117 111
query62 787 648 654 648
query63 322 293 300 293
query64 9115 2211 2695 2211
query65 3161 3111 3118 3111
query66 767 328 321 321
query67 15456 15160 15192 15160
query68 4544 550 569 550
query69 610 389 347 347
query70 1133 1108 1144 1108
query71 382 290 286 286
query72 7259 5565 5523 5523
query73 751 333 331 331
query74 5885 5529 5501 5501
query75 3421 2716 2727 2716
query76 2601 923 886 886
query77 596 305 323 305
query78 10742 8924 8880 8880
query79 3339 526 537 526
query80 1630 485 482 482
query81 590 220 226 220
query82 1029 144 136 136
query83 340 177 173 173
query84 278 89 89 89
query85 1385 317 310 310
query86 468 335 309 309
query87 3296 3116 3149 3116
query88 4464 2457 2472 2457
query89 506 406 397 397
query90 1812 205 202 202
query91 133 107 105 105
query92 59 51 51 51
query93 4719 521 511 511
query94 1174 222 215 215
query95 420 329 331 329
query96 637 276 351 276
query97 3205 3057 3039 3039
query98 224 201 196 196
query99 1517 1250 1251 1250
Total cold run time: 277889 ms
Total hot run time: 175975 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org