You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/10/09 12:50:22 UTC
[PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
morningman opened a new pull request, #68839:
URL: https://github.com/apache/doris/pull/68839
### What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary:
**In short.** This ports the test coverage of branch-4.1 #66717 (memory governance for external metadata caches) to master. master already has the feature (#67726), but none of the 4.x tests came along, and several admission rules are now unpinned. Only tests are added; no production code changes.
**Background.** #66717 bounds the estimated retained memory of external metadata caches with three limits: the FE-wide `external_meta_cache_max_weight`, the catalog property `meta.cache.max-weight`, and per-entry `meta.cache.<engine>.<entry>.max-weight`. master re-implemented it in #67726 on the connector cache framework instead of cherry-picking it, so the 4.x tests, written against fe-core classes master does not have (`ExternalMetaCacheBudgetManager`, `MetaCacheEntry`, `IcebergExternalMetaCache`, ...), could not be carried over.
| | branch-4.1 #66717 | master #67726 |
|---|---|---|
| Where it lives | fe-core `datasource/metacache` | `fe-connector-cache` plus each connector |
| Budget levels | global / catalog / entry | global / catalog / entry group (physical caches of one logical entry share it) |
| Cached values | soft values | strong values with explicit reservations |
| Budget short on admission | synchronously evict the entry's coldest value, then admit | reject this load, reclaim from peers in the background, admit on the next miss |
**The problem, and what it cost.** The admission logic itself is equivalent, but on master:
- `MetaCacheBudgetManager` has no test of its own; it is only reached through caches built on top of it. The three-level limits, clamping a catalog replayed from an FE with a larger global quota, the max(old, new) hand-off between two generations of one key, force release on close, and the rules for whom a peer reclamation may ask (only the requester's catalog when the catalog limit is short, only the same budget group when the group limit is short, queued requests collapsed to the largest one) could all regress without a red test.
- At the cache level, zero and incomplete estimates, disabled entries, oversized values, exact accounting when a key is replaced by a smaller or larger value, reclamation across catalogs when only the global quota is short, and failed refreshes are not pinned.
- At the connector level nothing checks which Hive and Iceberg caches a catalog or entry limit actually reaches, that invalidating a Paimon table gives its reservation back, or that statement copies of historical Iceberg snapshots stay apart from the cached generation.
**How this PR fixes it.** It adds those checks in master's terms:
| Test | What it pins |
|---|---|
| `MetaCacheBudgetManagerTest` (new, 12) | global / catalog / entry limits; 200 concurrent reservations never exceed the global quota; clamping to the smallest enclosing limit; weight parsing (binary units, heap percentages, rejected forms); idempotent release; replacement commit and rollback; closed budgets; close-time force release; peer reclamation coalescing and its catalog / group scoping; catalog and group bucket lifecycle |
| `MetaCacheWeightGovernanceTest` (+6) | zero or incomplete estimates are returned but not cached; a disabled entry never estimates; an oversized value does not evict cached ones; same-key replacement charges exactly the current generation; a catalog short only on the global quota gets it back from another catalog; a failed refresh keeps the cached value and its reservation |
| `HiveConnectorWeightGovernanceTest` (new, 2) | a catalog limit bounds the connector's file listing and partition view caches and closing the connector releases them; an entry limit bounds only its own entry |
| `PaimonMetaCacheCatalogTest` (+1) | invalidating a table releases its reservation and the reload is admitted again |
| `IcebergConnectorCacheTest` (+1) | which connector caches the catalog, manifest and partition view limits reach |
| `IcebergTableCacheTest` (+1) | statement copies of historical snapshots are not the cached generation's objects |
Not ported, because master deliberately differs and has its own tests for its choice: soft values, synchronous local eviction before admission, keeping the old value when a replacement is rejected, and Iceberg generation fencing. The `test_hive_meta_cache` message change ("is wrong" to "must be >= -1") does not apply because master keeps the legacy message.
**Results.** All new tests pass on current master. Gaps in product behavior found while comparing the two implementations (for example the system table columns 4.x exposes and master does not) are left to separate PRs.
Original author: @CalvinKirs
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `run-fe-ut.sh --run` on `MetaCacheBudgetManagerTest`, `MetaCacheWeightGovernanceTest`, `HiveConnectorWeightGovernanceTest`, `PaimonMetaCacheCatalogTest`, `IcebergConnectorCacheTest`, `IcebergTableCacheTest`: 110 tests, 0 failures
- `mvn checkstyle:check` on the four touched modules: 0 violations
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6082592806
<details>
<summary>TPC-DS: <b>Total hot run time: 151493 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d76b69ce222e25f81491146f11209846de0840b, data reload: false
query5 4307 613 431 431
query6 422 208 187 187
query7 4809 443 227 227
query8 316 181 166 166
query9 8763 3925 3903 3903
query10 471 303 257 257
query11 5948 3420 3176 3176
query12 150 92 87 87
query13 1260 431 310 310
query14 6536 4767 4452 4452
query14_1 4228 4215 4203 4203
query15 206 199 186 186
query16 936 440 421 421
query17 866 654 536 536
query18 2418 445 320 320
query19 197 163 136 136
query20 81 80 80 80
query21 207 132 111 111
query22 12932 13006 12777 12777
query23 13143 12571 12234 12234
query23_1 12221 12273 12141 12141
query24 7095 835 466 466
query24_1 466 470 475 470
query25 530 409 361 361
query26 1243 264 141 141
query27 2754 446 266 266
query28 4615 1912 1963 1912
query29 1542 595 457 457
query30 300 221 185 185
query31 890 767 649 649
query32 142 88 95 88
query33 503 308 249 249
query34 900 849 498 498
query35 757 767 646 646
query36 815 804 729 729
query37 127 105 92 92
query38 1782 1739 1719 1719
query39 707 680 674 674
query39_1 680 656 678 656
query40 227 112 95 95
query41 74 68 69 68
query42 89 86 87 86
query43 362 384 335 335
query44 1316 694 686 686
query45 183 184 160 160
query46 838 961 561 561
query47 3017 3059 2904 2904
query48 321 310 211 211
query49 558 408 281 281
query50 670 265 213 213
query51 10511 10571 10384 10384
query52 77 85 67 67
query53 187 199 146 146
query54 231 218 198 198
query55 67 66 62 62
query56 240 222 197 197
query57 1606 1593 1573 1573
query58 266 250 267 250
query59 2355 2355 2113 2113
query60 280 239 213 213
query61 135 164 144 144
query62 389 345 286 286
query63 197 156 166 156
query64 2734 965 763 763
query65 3439 3358 3365 3358
query66 1759 410 297 297
query67 20298 20164 20034 20034
query68 3202 1004 608 608
query69 416 282 242 242
query70 937 831 820 820
query71 318 236 209 209
query72 2796 2554 2273 2273
query73 543 527 287 287
query74 4557 4440 4274 4274
query75 2341 2288 1967 1967
query76 2400 999 584 584
query77 361 370 290 290
query78 8989 8955 8496 8496
query79 972 911 508 508
query80 692 410 329 329
query81 558 327 273 273
query82 573 140 109 109
query83 276 199 174 174
query84 316 117 94 94
query85 880 428 362 362
query86 327 238 235 235
query87 2025 1968 1810 1810
query88 3627 2693 2688 2688
query89 333 295 253 253
query90 1782 177 178 177
query91 164 151 120 120
query92 94 87 87 87
query93 1028 1031 560 560
query94 489 288 274 274
query95 584 330 383 330
query96 672 541 235 235
query97 2451 2406 2311 2311
query98 153 147 144 144
query99 778 764 640 640
Total cold run time: 234116 ms
Total hot run time: 151493 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6081340665
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6087595599
<details>
<summary>TPC-H: <b>Total hot run time: 28716 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 742c58b19b104382a449f16bac1635a01b8e063c, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17642 5615 5561 5561
q2 2092 315 256 256
q3 10270 1369 822 822
q4 4672 476 353 353
q5 7418 788 533 533
q6 184 184 154 154
q7 759 790 592 592
q8 9248 1320 1035 1035
q9 5618 4391 4406 4391
q10 6806 1287 1009 1009
q11 414 247 228 228
q12 626 394 277 277
q13 18095 3016 2375 2375
q14 280 264 242 242
q15 q16 731 722 680 680
q17 1159 721 566 566
q18 7030 6177 6164 6164
q19 1111 954 599 599
q20 386 335 218 218
q21 5561 2371 2549 2371
q22 392 338 290 290
Total cold run time: 100494 ms
Total hot run time: 28716 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6509 6483 6500 6483
q2 723 588 553 553
q3 5560 5711 5307 5307
q4 2179 2204 1622 1622
q5 5737 5852 5792 5792
q6 255 198 151 151
q7 2290 1969 1825 1825
q8 3063 2734 2736 2734
q9 8111 7744 7724 7724
q10 3655 3570 3202 3202
q11 595 396 375 375
q12 682 702 486 486
q13 2714 3062 2369 2369
q14 280 290 259 259
q15 q16 667 690 599 599
q17 7786 6946 6863 6863
q18 13091 12290 13019 12290
q19 914 771 766 766
q20 2185 2162 1924 1924
q21 5789 4680 4795 4680
q22 498 442 412 412
Total cold run time: 73283 ms
Total hot run time: 66416 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6094279191
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/77)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68839_3506202c47e3aa4ad188b12227c64afa8f83f029_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68839_3506202c47e3aa4ad188b12227c64afa8f83f029_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6087824426
<details>
<summary>ClickBench: <b>Total hot run time: 24.37 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 742c58b19b104382a449f16bac1635a01b8e063c, data reload: false
query1 0.01 0.00 0.01
query2 0.11 0.05 0.05
query3 0.26 0.11 0.12
query4 1.60 0.12 0.12
query5 0.27 0.25 0.25
query6 1.18 0.67 0.68
query7 0.03 0.00 0.00
query8 0.06 0.03 0.03
query9 0.40 0.35 0.35
query10 0.57 0.60 0.56
query11 0.20 0.14 0.14
query12 0.17 0.14 0.15
query13 0.50 0.49 0.50
query14 0.97 0.97 0.96
query15 0.62 0.60 0.60
query16 0.33 0.36 0.31
query17 1.13 1.15 1.13
query18 0.23 0.21 0.20
query19 2.12 2.01 1.93
query20 0.02 0.01 0.01
query21 15.43 0.24 0.14
query22 4.61 0.05 0.05
query23 16.26 0.33 0.12
query24 3.33 0.65 0.39
query25 0.11 0.05 0.06
query26 1.08 0.26 0.15
query27 0.04 0.03 0.03
query28 1.65 0.63 0.44
query29 12.49 4.13 3.26
query30 0.28 0.14 0.13
query31 2.77 0.60 0.34
query32 3.24 0.61 0.54
query33 3.10 3.18 3.26
query34 15.49 4.03 3.39
query35 3.36 3.32 3.31
query36 0.53 0.48 0.46
query37 0.10 0.06 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.18 0.14 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.02
query43 0.04 0.03 0.04
Total cold run time: 95.08 s
Total hot run time: 24.37 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6093171047
<details>
<summary>TPC-H: <b>Total hot run time: 29102 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3506202c47e3aa4ad188b12227c64afa8f83f029, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17592 5675 5579 5579
q2 2100 308 269 269
q3 10251 1403 834 834
q4 4682 472 341 341
q5 7432 796 528 528
q6 190 191 156 156
q7 757 800 610 610
q8 9230 1382 1147 1147
q9 5768 4447 4477 4447
q10 6814 1310 1036 1036
q11 434 254 238 238
q12 630 403 284 284
q13 18058 3133 2392 2392
q14 275 262 251 251
q15 q16 745 734 673 673
q17 1168 741 547 547
q18 7037 6245 6181 6181
q19 1101 951 584 584
q20 402 351 226 226
q21 5626 2500 2669 2500
q22 420 348 279 279
Total cold run time: 100712 ms
Total hot run time: 29102 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6470 6482 6439 6439
q2 750 612 572 572
q3 5494 5700 5263 5263
q4 2309 2208 1589 1589
q5 5627 5697 5810 5697
q6 248 196 150 150
q7 2331 1958 1895 1895
q8 3135 2770 2790 2770
q9 8210 8060 7771 7771
q10 3681 3587 3227 3227
q11 615 418 373 373
q12 667 705 499 499
q13 2762 3089 2388 2388
q14 287 297 262 262
q15 q16 694 676 602 602
q17 7787 6961 6955 6955
q18 13226 12287 13155 12287
q19 916 784 780 780
q20 2190 2157 1967 1967
q21 5791 4821 4808 4808
q22 506 456 408 408
Total cold run time: 73696 ms
Total hot run time: 66702 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68839:
URL: https://github.com/apache/doris/pull/68839#discussion_r4233279888
##########
fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheBudgetManagerTest.java:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.cache;
+
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.AdmissionReservation;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.EntryBudget;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.ReservationReplacement;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Budget arithmetic of {@link MetaCacheBudgetManager} without a cache on top: the global, catalog and
+ * entry-group limits, the reservation hand-off between two generations of one key, release on close,
+ * and which peers a reclamation request may ask.
+ */
+class MetaCacheBudgetManagerTest {
+ private static final OptionalLong NONE = OptionalLong.empty();
+
+ @Test
+ void reservationsRespectGlobalCatalogAndEntryLimits() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget file = manager.createEntryBudget(
+ 1L, "hive", "file", "file", OptionalLong.of(80L), OptionalLong.of(60L));
+ EntryBudget partitionView = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", OptionalLong.of(80L), OptionalLong.of(50L));
+
+ AdmissionReservation fileReservation = file.tryReserve(60L).get();
+ Assertions.assertFalse(partitionView.tryReserve(30L).isPresent(),
+ "the catalog limit of 80 is already 60 used");
+ AdmissionReservation viewReservation = partitionView.tryReserve(20L).get();
+ Assertions.assertEquals(80L, manager.getGlobalUsedWeight());
+ Assertions.assertFalse(partitionView.tryReplace(viewReservation, 30L).isPresent());
+
+ fileReservation.release();
+ ReservationReplacement grown = partitionView.tryReplace(viewReservation, 30L).get();
+ grown.commit();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ Assertions.assertEquals(30L, partitionView.getUsedWeight());
+
+ grown.current().release();
+ Assertions.assertFalse(partitionView.tryReserve(51L).isPresent(),
+ "the entry limit of 50 applies even when the catalog and global budgets have room");
+ file.close();
+ partitionView.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void concurrentReservationsNeverExceedTheGlobalLimit() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "manifest", "manifest", NONE, NONE);
+ ExecutorService executor = Executors.newFixedThreadPool(8);
+ CountDownLatch start = new CountDownLatch(1);
+ List<AdmissionReservation> reservations = Collections.synchronizedList(new ArrayList<>());
+ try {
+ for (int i = 0; i < 200; i++) {
+ executor.submit(() -> {
Review Comment:
Fixed in 742c58b19b1: the test keeps the 200 futures and calls `get()` on each before checking the totals, so an attempt that threw now fails the test instead of looking like a refused reservation.
##########
fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheBudgetManagerTest.java:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.cache;
+
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.AdmissionReservation;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.EntryBudget;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.ReservationReplacement;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Budget arithmetic of {@link MetaCacheBudgetManager} without a cache on top: the global, catalog and
+ * entry-group limits, the reservation hand-off between two generations of one key, release on close,
+ * and which peers a reclamation request may ask.
+ */
+class MetaCacheBudgetManagerTest {
+ private static final OptionalLong NONE = OptionalLong.empty();
+
+ @Test
+ void reservationsRespectGlobalCatalogAndEntryLimits() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget file = manager.createEntryBudget(
+ 1L, "hive", "file", "file", OptionalLong.of(80L), OptionalLong.of(60L));
+ EntryBudget partitionView = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", OptionalLong.of(80L), OptionalLong.of(50L));
+
+ AdmissionReservation fileReservation = file.tryReserve(60L).get();
+ Assertions.assertFalse(partitionView.tryReserve(30L).isPresent(),
+ "the catalog limit of 80 is already 60 used");
+ AdmissionReservation viewReservation = partitionView.tryReserve(20L).get();
+ Assertions.assertEquals(80L, manager.getGlobalUsedWeight());
+ Assertions.assertFalse(partitionView.tryReplace(viewReservation, 30L).isPresent());
+
+ fileReservation.release();
+ ReservationReplacement grown = partitionView.tryReplace(viewReservation, 30L).get();
+ grown.commit();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ Assertions.assertEquals(30L, partitionView.getUsedWeight());
+
+ grown.current().release();
+ Assertions.assertFalse(partitionView.tryReserve(51L).isPresent(),
+ "the entry limit of 50 applies even when the catalog and global budgets have room");
+ file.close();
+ partitionView.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void concurrentReservationsNeverExceedTheGlobalLimit() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "manifest", "manifest", NONE, NONE);
+ ExecutorService executor = Executors.newFixedThreadPool(8);
+ CountDownLatch start = new CountDownLatch(1);
+ List<AdmissionReservation> reservations = Collections.synchronizedList(new ArrayList<>());
+ try {
+ for (int i = 0; i < 200; i++) {
+ executor.submit(() -> {
+ await(start);
+ budget.tryReserve(1L).ifPresent(reservations::add);
+ });
+ }
+ start.countDown();
+ executor.shutdown();
+ Assertions.assertTrue(executor.awaitTermination(10L, TimeUnit.SECONDS));
+ Assertions.assertEquals(100, reservations.size());
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ } finally {
+ executor.shutdownNow();
+ reservations.forEach(AdmissionReservation::release);
+ budget.close();
+ }
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void entryBudgetIsClampedToTheSmallestEnclosingLimit() {
+ // A catalog created on an FE with a larger global quota replays here with limits above ours.
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget replayed = manager.createEntryBudget(
+ 1L, "iceberg", "table", "table", OptionalLong.of(400L), OptionalLong.of(300L));
+ Assertions.assertEquals(100L, replayed.getEffectiveMaxWeight());
+ AdmissionReservation reservation = replayed.tryReserve(100L).get();
+ Assertions.assertFalse(replayed.tryReserve(1L).isPresent());
+ reservation.release();
+ replayed.close();
+
+ EntryBudget entryAboveCatalog = manager.createEntryBudget(
+ 2L, "iceberg", "table", "table", OptionalLong.of(40L), OptionalLong.of(60L));
+ Assertions.assertEquals(40L, entryAboveCatalog.getEffectiveMaxWeight());
+ Assertions.assertFalse(entryAboveCatalog.tryReserve(41L).isPresent());
+ entryAboveCatalog.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void weightParsingAcceptsBinaryUnitsAndHeapPercentages() {
+ long heap = 4L * 1024L * 1024L * 1024L;
+ String key = "external_meta_cache_max_weight";
+ Assertions.assertEquals(1L, CacheSpec.parseWeight("1", key, true, heap));
+ Assertions.assertEquals(2L * 1024L * 1024L, CacheSpec.parseWeight(" 2 mb ", key, true, heap));
+ Assertions.assertEquals(heap / 4L, CacheSpec.parseWeight("25%", key, true, heap));
+ Assertions.assertEquals(heap / 8L, CacheSpec.parseWeight("12.5%", key, true, heap));
+ // 0 disables the global quota; rejecting 0% is the caller's job (ExternalMetaCacheMgr).
+ Assertions.assertEquals(0L, CacheSpec.parseWeight("0", key, true, heap));
+ Assertions.assertEquals(0L, CacheSpec.parseWeight("0%", key, true, heap));
+
+ for (String invalid : new String[] {"", "1.5GB", "101%", "-1%", "-1", "8EB", "1PB000",
+ "8192PB", "999999999999999999PB"}) {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> CacheSpec.parseWeight(invalid, key, true, heap), invalid);
+ }
+ IllegalArgumentException catalogPercentage = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> CacheSpec.parseWeight("10%", "meta.cache.max-weight", false, heap));
+ Assertions.assertEquals("Invalid cache weight for 'meta.cache.max-weight': 10%",
+ catalogPercentage.getMessage());
+ }
+
+ @Test
+ void releasingAReservationTwiceReleasesItOnce() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget first = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ EntryBudget second = manager.createEntryBudget(1L, "hive", "file", "file", NONE, NONE);
+ AdmissionReservation released = first.tryReserve(40L).get();
+ AdmissionReservation kept = second.tryReserve(30L).get();
+
+ released.release();
+ released.release();
+
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight(),
+ "a repeated release must not take bytes from another reservation");
+ Assertions.assertEquals(0L, released.getBytes());
+ kept.release();
+ first.close();
+ second.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void replacementHoldsTheLargerGenerationUntilCommitOrRollback() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "table", "table", NONE, NONE);
+ AdmissionReservation first = budget.tryReserve(30L).get();
+
+ ReservationReplacement grow = budget.tryReplace(first, 50L).get();
+ Assertions.assertEquals(50L, manager.getGlobalUsedWeight(), "max(old, new) is held, never old + new");
+ Assertions.assertFalse(budget.tryReplace(first, 10L).isPresent(),
+ "a generation that is being replaced can not be replaced again");
+ grow.commit();
+ AdmissionReservation second = grow.current();
+ Assertions.assertEquals(50L, second.getBytes());
+
+ ReservationReplacement shrink = budget.tryReplace(second, 10L).get();
+ Assertions.assertEquals(50L, manager.getGlobalUsedWeight(),
+ "the old generation stays charged until the new one is published");
+ shrink.commit();
+ Assertions.assertEquals(10L, manager.getGlobalUsedWeight());
+
+ AdmissionReservation third = shrink.current();
+ ReservationReplacement abandoned = budget.tryReplace(third, 40L).get();
+ Assertions.assertEquals(40L, manager.getGlobalUsedWeight());
+ abandoned.rollback();
+ Assertions.assertEquals(10L, manager.getGlobalUsedWeight());
+ Assertions.assertThrows(IllegalStateException.class, abandoned::commit);
+ third.release();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight(), "rollback reactivates the previous generation");
+ budget.close();
+ }
+
+ @Test
+ void closedBudgetRejectsNewReservationsAndReplacements() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget stale = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation zeroBytes = stale.tryReserve(0L).get();
+
+ stale.close();
+ stale.close();
+
+ Assertions.assertFalse(stale.tryReserve(1L).isPresent());
+ Assertions.assertFalse(stale.tryReplace(zeroBytes, 1L).isPresent());
+ zeroBytes.release();
+
+ EntryBudget replacement = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation full = replacement.tryReserve(100L).get();
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ full.release();
+ replacement.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void closeForceReleasesOutstandingReservations() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget closing = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ EntryBudget sibling = manager.createEntryBudget(1L, "hive", "file", "file", NONE, NONE);
+ AdmissionReservation leaked = closing.tryReserve(40L).get();
+ AdmissionReservation live = sibling.tryReserve(30L).get();
+
+ closing.close();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ leaked.release();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight(),
+ "a late release from a closed owner must not be subtracted twice");
+ Assertions.assertEquals(30L, sibling.getUsedWeight());
+
+ EntryBudget reopened = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation reReserved = reopened.tryReserve(70L).get();
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ reReserved.release();
+ live.release();
+ reopened.close();
+ sibling.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void peerReclaimCoalescesQueuedRequestsToTheLargestAdmission() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget owner = manager.createEntryBudget(1L, "iceberg", "partition", "partition", NONE, NONE);
+ EntryBudget requester = manager.createEntryBudget(2L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation reservation = owner.tryReserve(100L).get();
+ CountDownLatch firstReclaimStarted = new CountDownLatch(1);
+ CountDownLatch releaseFirstReclaim = new CountDownLatch(1);
+ CountDownLatch secondReclaimFinished = new CountDownLatch(1);
+ AtomicInteger invocations = new AtomicInteger();
+ List<Long> targets = Collections.synchronizedList(new ArrayList<>());
+ owner.setReclaimer(target -> {
+ targets.add(target);
+ if (invocations.getAndIncrement() == 0) {
+ firstReclaimStarted.countDown();
+ await(releaseFirstReclaim);
+ } else {
+ secondReclaimFinished.countDown();
+ }
+ return 0L;
+ });
+ try {
+ requester.requestPeerReclaim(10L);
+ Assertions.assertTrue(firstReclaimStarted.await(10L, TimeUnit.SECONDS));
+
+ requester.requestPeerReclaim(10L);
+ requester.requestPeerReclaim(20L);
+ requester.requestPeerReclaim(15L);
+ releaseFirstReclaim.countDown();
+
+ Assertions.assertTrue(secondReclaimFinished.await(10L, TimeUnit.SECONDS));
+ Assertions.assertEquals(Arrays.asList(10L, 20L), new ArrayList<>(targets),
+ "misses queued behind a running reclamation collapse into one request for the largest");
+ } finally {
+ releaseFirstReclaim.countDown();
+ reservation.release();
+ owner.close();
+ requester.close();
+ }
+ }
+
+ @Test
+ void catalogDeficitNeverAsksAnotherCatalog() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(200L));
+ EntryBudget sibling = manager.createEntryBudget(
+ 1L, "iceberg", "partition", "partition", OptionalLong.of(100L), NONE);
+ EntryBudget requester = manager.createEntryBudget(
+ 1L, "iceberg", "manifest", "manifest", OptionalLong.of(100L), NONE);
+ EntryBudget otherCatalog = manager.createEntryBudget(
+ 2L, "paimon", "partition_view", "partition_view", OptionalLong.of(100L), NONE);
+ AdmissionReservation siblingReservation = sibling.tryReserve(100L).get();
+ // 150 of the global 200 is used, so the request below is short only at the catalog level.
+ AdmissionReservation otherReservation = otherCatalog.tryReserve(50L).get();
+ CountDownLatch siblingAskedTwice = new CountDownLatch(2);
+ List<Long> siblingTargets = Collections.synchronizedList(new ArrayList<>());
+ AtomicInteger otherCatalogAsks = new AtomicInteger();
+ sibling.setReclaimer(target -> {
+ siblingTargets.add(target);
+ siblingAskedTwice.countDown();
+ return 0L;
+ });
+ otherCatalog.setReclaimer(target -> {
+ otherCatalogAsks.incrementAndGet();
+ return 0L;
+ });
+ try {
+ requester.requestPeerReclaim(20L);
+ awaitCount(siblingTargets, 1);
+ // The reclaim executor runs one drain at a time, so the second request is served only after
+ // the first drain has walked every candidate.
+ requester.requestPeerReclaim(20L);
+
+ Assertions.assertTrue(siblingAskedTwice.await(10L, TimeUnit.SECONDS));
+ Assertions.assertEquals(Long.valueOf(20L), siblingTargets.get(0));
+ Assertions.assertEquals(0, otherCatalogAsks.get(),
+ "only this catalog's caches can relieve a catalog-level deficit");
+ } finally {
+ siblingReservation.release();
+ otherReservation.release();
+ sibling.close();
+ requester.close();
+ otherCatalog.close();
+ }
+ }
+
+ @Test
+ void groupDeficitOnlyAsksPhysicalCachesOfTheSameGroup() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(NONE);
+ EntryBudget groupPeer = manager.createEntryBudget(
+ 1L, "iceberg", "mvcc-partition-view", "partition_view", NONE, OptionalLong.of(100L));
+ EntryBudget requester = manager.createEntryBudget(
+ 1L, "iceberg", "list-partitions-view", "partition_view", NONE, OptionalLong.of(100L));
+ EntryBudget otherGroup = manager.createEntryBudget(
+ 1L, "iceberg", "manifest", "manifest", NONE, OptionalLong.of(500L));
+ AdmissionReservation peerReservation = groupPeer.tryReserve(100L).get();
+ AdmissionReservation otherReservation = otherGroup.tryReserve(400L).get();
+ CountDownLatch peerAskedTwice = new CountDownLatch(2);
+ List<Long> peerTargets = Collections.synchronizedList(new ArrayList<>());
+ AtomicInteger otherGroupAsks = new AtomicInteger();
+ groupPeer.setReclaimer(target -> {
+ peerTargets.add(target);
+ peerAskedTwice.countDown();
+ return 0L;
+ });
+ otherGroup.setReclaimer(target -> {
+ otherGroupAsks.incrementAndGet();
+ return 0L;
+ });
+ try {
+ requester.requestPeerReclaim(20L);
+ awaitCount(peerTargets, 1);
+ requester.requestPeerReclaim(20L);
+
+ Assertions.assertTrue(peerAskedTwice.await(10L, TimeUnit.SECONDS));
+ Assertions.assertEquals(Long.valueOf(20L), peerTargets.get(0));
+ Assertions.assertEquals(0, otherGroupAsks.get(),
+ "the larger cache of another group can not relieve this group's shared limit");
+ } finally {
+ peerReservation.release();
+ otherReservation.release();
+ groupPeer.close();
+ requester.close();
+ otherGroup.close();
+ }
+ }
+
+ @Test
+ void catalogAndGroupBucketsLiveExactlyAsLongAsTheirEntries() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(1L << 20));
+ List<EntryBudget> budgets = new ArrayList<>();
+ for (long catalogId = 1L; catalogId <= 3L; catalogId++) {
+ for (String entry : new String[] {"paimon-table", "partition_view"}) {
Review Comment:
Fixed in 742c58b19b1: added `sharedEntryGroupKeepsItsLimitUntilItsLastPhysicalCacheCloses`. Two physical caches share the `partition_view` group. One holds 60 of the group's 100 bytes; the other is closed and re-created and can then take only 40. The group bucket goes away only when the last of them closes. `MetaCacheBudgetManager#close` already counts live members per group bucket, so this is test-only; making `close` drop the bucket unconditionally turns the new test red.
##########
fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java:
##########
@@ -226,6 +229,43 @@ private static Table fakeTable(String name) {
PartitionSpec.unpartitioned(), "s3://b/" + name, Collections.emptyMap());
}
+ @Test
+ public void weightLimitsReachEveryConnectorOwnedCache() throws Exception {
Review Comment:
Fixed in 742c58b19b1: the catalog-limit check now also builds a REST vended-credentials catalog, the only flavor that creates `iceberg-comment`, and asserts that it reports the 1 MB catalog limit. Building the comment cache outside the catalog's governed owner turns it red.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6082679509
<details>
<summary>ClickBench: <b>Total hot run time: 24.34 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d76b69ce222e25f81491146f11209846de0840b, data reload: false
query1 0.00 0.00 0.00
query2 0.09 0.05 0.05
query3 0.26 0.12 0.12
query4 1.61 0.12 0.13
query5 0.24 0.25 0.26
query6 1.16 0.68 0.68
query7 0.04 0.01 0.00
query8 0.06 0.04 0.04
query9 0.41 0.35 0.35
query10 0.57 0.55 0.62
query11 0.20 0.14 0.15
query12 0.18 0.15 0.15
query13 0.48 0.48 0.50
query14 0.97 0.96 0.97
query15 0.63 0.59 0.60
query16 0.34 0.34 0.35
query17 1.11 1.12 1.16
query18 0.22 0.20 0.21
query19 2.12 2.01 1.97
query20 0.02 0.01 0.01
query21 15.52 0.28 0.13
query22 4.36 0.05 0.05
query23 16.27 0.31 0.12
query24 3.25 0.55 0.37
query25 0.13 0.06 0.07
query26 1.08 0.25 0.16
query27 0.06 0.03 0.03
query28 1.54 0.60 0.43
query29 12.55 4.03 3.24
query30 0.28 0.15 0.13
query31 2.78 0.58 0.33
query32 3.21 0.62 0.51
query33 3.12 3.16 3.21
query34 15.48 3.99 3.40
query35 3.32 3.35 3.34
query36 0.56 0.43 0.41
query37 0.10 0.06 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.04
query40 0.16 0.14 0.15
query41 0.09 0.03 0.03
query42 0.04 0.02 0.04
query43 0.04 0.03 0.03
Total cold run time: 94.74 s
Total hot run time: 24.34 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68839:
URL: https://github.com/apache/doris/pull/68839#discussion_r4230503115
##########
fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheBudgetManagerTest.java:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.cache;
+
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.AdmissionReservation;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.EntryBudget;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.ReservationReplacement;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Budget arithmetic of {@link MetaCacheBudgetManager} without a cache on top: the global, catalog and
+ * entry-group limits, the reservation hand-off between two generations of one key, release on close,
+ * and which peers a reclamation request may ask.
+ */
+class MetaCacheBudgetManagerTest {
+ private static final OptionalLong NONE = OptionalLong.empty();
+
+ @Test
+ void reservationsRespectGlobalCatalogAndEntryLimits() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget file = manager.createEntryBudget(
+ 1L, "hive", "file", "file", OptionalLong.of(80L), OptionalLong.of(60L));
+ EntryBudget partitionView = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", OptionalLong.of(80L), OptionalLong.of(50L));
+
+ AdmissionReservation fileReservation = file.tryReserve(60L).get();
+ Assertions.assertFalse(partitionView.tryReserve(30L).isPresent(),
+ "the catalog limit of 80 is already 60 used");
+ AdmissionReservation viewReservation = partitionView.tryReserve(20L).get();
+ Assertions.assertEquals(80L, manager.getGlobalUsedWeight());
+ Assertions.assertFalse(partitionView.tryReplace(viewReservation, 30L).isPresent());
+
+ fileReservation.release();
+ ReservationReplacement grown = partitionView.tryReplace(viewReservation, 30L).get();
+ grown.commit();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ Assertions.assertEquals(30L, partitionView.getUsedWeight());
+
+ grown.current().release();
+ Assertions.assertFalse(partitionView.tryReserve(51L).isPresent(),
+ "the entry limit of 50 applies even when the catalog and global budgets have room");
+ file.close();
+ partitionView.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void concurrentReservationsNeverExceedTheGlobalLimit() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "manifest", "manifest", NONE, NONE);
+ ExecutorService executor = Executors.newFixedThreadPool(8);
+ CountDownLatch start = new CountDownLatch(1);
+ List<AdmissionReservation> reservations = Collections.synchronizedList(new ArrayList<>());
+ try {
+ for (int i = 0; i < 200; i++) {
+ executor.submit(() -> {
Review Comment:
[P2] Check every submitted reservation result. `submit` keeps worker exceptions in Futures, but this test discards them. If 100 calls reserve a byte and the other 100 throw instead of returning an empty Optional, termination, reservation count, and global weight all still pass. Collect the Futures and call `get()` on each before asserting the admission totals.
##########
fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java:
##########
@@ -226,6 +229,43 @@ private static Table fakeTable(String name) {
PartitionSpec.unpartitioned(), "s3://b/" + name, Collections.emptyMap());
}
+ @Test
+ public void weightLimitsReachEveryConnectorOwnedCache() throws Exception {
Review Comment:
[P2] Include the REST comment cache in the catalog-limit check. These three calls construct plain catalogs, so `iceberg-comment` is never registered and the fixed list cannot catch an unbounded comment cache. `IcebergConnector` builds it for REST vended-credential catalogs through the same owner. Add that configuration and assert the comment entry receives the catalog max weight.
##########
fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheBudgetManagerTest.java:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.cache;
+
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.AdmissionReservation;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.EntryBudget;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.ReservationReplacement;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Budget arithmetic of {@link MetaCacheBudgetManager} without a cache on top: the global, catalog and
+ * entry-group limits, the reservation hand-off between two generations of one key, release on close,
+ * and which peers a reclamation request may ask.
+ */
+class MetaCacheBudgetManagerTest {
+ private static final OptionalLong NONE = OptionalLong.empty();
+
+ @Test
+ void reservationsRespectGlobalCatalogAndEntryLimits() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget file = manager.createEntryBudget(
+ 1L, "hive", "file", "file", OptionalLong.of(80L), OptionalLong.of(60L));
+ EntryBudget partitionView = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", OptionalLong.of(80L), OptionalLong.of(50L));
+
+ AdmissionReservation fileReservation = file.tryReserve(60L).get();
+ Assertions.assertFalse(partitionView.tryReserve(30L).isPresent(),
+ "the catalog limit of 80 is already 60 used");
+ AdmissionReservation viewReservation = partitionView.tryReserve(20L).get();
+ Assertions.assertEquals(80L, manager.getGlobalUsedWeight());
+ Assertions.assertFalse(partitionView.tryReplace(viewReservation, 30L).isPresent());
+
+ fileReservation.release();
+ ReservationReplacement grown = partitionView.tryReplace(viewReservation, 30L).get();
+ grown.commit();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ Assertions.assertEquals(30L, partitionView.getUsedWeight());
+
+ grown.current().release();
+ Assertions.assertFalse(partitionView.tryReserve(51L).isPresent(),
+ "the entry limit of 50 applies even when the catalog and global budgets have room");
+ file.close();
+ partitionView.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void concurrentReservationsNeverExceedTheGlobalLimit() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "manifest", "manifest", NONE, NONE);
+ ExecutorService executor = Executors.newFixedThreadPool(8);
+ CountDownLatch start = new CountDownLatch(1);
+ List<AdmissionReservation> reservations = Collections.synchronizedList(new ArrayList<>());
+ try {
+ for (int i = 0; i < 200; i++) {
+ executor.submit(() -> {
+ await(start);
+ budget.tryReserve(1L).ifPresent(reservations::add);
+ });
+ }
+ start.countDown();
+ executor.shutdown();
+ Assertions.assertTrue(executor.awaitTermination(10L, TimeUnit.SECONDS));
+ Assertions.assertEquals(100, reservations.size());
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ } finally {
+ executor.shutdownNow();
+ reservations.forEach(AdmissionReservation::release);
+ budget.close();
+ }
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void entryBudgetIsClampedToTheSmallestEnclosingLimit() {
+ // A catalog created on an FE with a larger global quota replays here with limits above ours.
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget replayed = manager.createEntryBudget(
+ 1L, "iceberg", "table", "table", OptionalLong.of(400L), OptionalLong.of(300L));
+ Assertions.assertEquals(100L, replayed.getEffectiveMaxWeight());
+ AdmissionReservation reservation = replayed.tryReserve(100L).get();
+ Assertions.assertFalse(replayed.tryReserve(1L).isPresent());
+ reservation.release();
+ replayed.close();
+
+ EntryBudget entryAboveCatalog = manager.createEntryBudget(
+ 2L, "iceberg", "table", "table", OptionalLong.of(40L), OptionalLong.of(60L));
+ Assertions.assertEquals(40L, entryAboveCatalog.getEffectiveMaxWeight());
+ Assertions.assertFalse(entryAboveCatalog.tryReserve(41L).isPresent());
+ entryAboveCatalog.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void weightParsingAcceptsBinaryUnitsAndHeapPercentages() {
+ long heap = 4L * 1024L * 1024L * 1024L;
+ String key = "external_meta_cache_max_weight";
+ Assertions.assertEquals(1L, CacheSpec.parseWeight("1", key, true, heap));
+ Assertions.assertEquals(2L * 1024L * 1024L, CacheSpec.parseWeight(" 2 mb ", key, true, heap));
+ Assertions.assertEquals(heap / 4L, CacheSpec.parseWeight("25%", key, true, heap));
+ Assertions.assertEquals(heap / 8L, CacheSpec.parseWeight("12.5%", key, true, heap));
+ // 0 disables the global quota; rejecting 0% is the caller's job (ExternalMetaCacheMgr).
+ Assertions.assertEquals(0L, CacheSpec.parseWeight("0", key, true, heap));
+ Assertions.assertEquals(0L, CacheSpec.parseWeight("0%", key, true, heap));
+
+ for (String invalid : new String[] {"", "1.5GB", "101%", "-1%", "-1", "8EB", "1PB000",
+ "8192PB", "999999999999999999PB"}) {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> CacheSpec.parseWeight(invalid, key, true, heap), invalid);
+ }
+ IllegalArgumentException catalogPercentage = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> CacheSpec.parseWeight("10%", "meta.cache.max-weight", false, heap));
+ Assertions.assertEquals("Invalid cache weight for 'meta.cache.max-weight': 10%",
+ catalogPercentage.getMessage());
+ }
+
+ @Test
+ void releasingAReservationTwiceReleasesItOnce() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget first = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ EntryBudget second = manager.createEntryBudget(1L, "hive", "file", "file", NONE, NONE);
+ AdmissionReservation released = first.tryReserve(40L).get();
+ AdmissionReservation kept = second.tryReserve(30L).get();
+
+ released.release();
+ released.release();
+
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight(),
+ "a repeated release must not take bytes from another reservation");
+ Assertions.assertEquals(0L, released.getBytes());
+ kept.release();
+ first.close();
+ second.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void replacementHoldsTheLargerGenerationUntilCommitOrRollback() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "table", "table", NONE, NONE);
+ AdmissionReservation first = budget.tryReserve(30L).get();
+
+ ReservationReplacement grow = budget.tryReplace(first, 50L).get();
+ Assertions.assertEquals(50L, manager.getGlobalUsedWeight(), "max(old, new) is held, never old + new");
+ Assertions.assertFalse(budget.tryReplace(first, 10L).isPresent(),
+ "a generation that is being replaced can not be replaced again");
+ grow.commit();
+ AdmissionReservation second = grow.current();
+ Assertions.assertEquals(50L, second.getBytes());
+
+ ReservationReplacement shrink = budget.tryReplace(second, 10L).get();
+ Assertions.assertEquals(50L, manager.getGlobalUsedWeight(),
+ "the old generation stays charged until the new one is published");
+ shrink.commit();
+ Assertions.assertEquals(10L, manager.getGlobalUsedWeight());
+
+ AdmissionReservation third = shrink.current();
+ ReservationReplacement abandoned = budget.tryReplace(third, 40L).get();
+ Assertions.assertEquals(40L, manager.getGlobalUsedWeight());
+ abandoned.rollback();
+ Assertions.assertEquals(10L, manager.getGlobalUsedWeight());
+ Assertions.assertThrows(IllegalStateException.class, abandoned::commit);
+ third.release();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight(), "rollback reactivates the previous generation");
+ budget.close();
+ }
+
+ @Test
+ void closedBudgetRejectsNewReservationsAndReplacements() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget stale = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation zeroBytes = stale.tryReserve(0L).get();
+
+ stale.close();
+ stale.close();
+
+ Assertions.assertFalse(stale.tryReserve(1L).isPresent());
+ Assertions.assertFalse(stale.tryReplace(zeroBytes, 1L).isPresent());
+ zeroBytes.release();
+
+ EntryBudget replacement = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation full = replacement.tryReserve(100L).get();
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ full.release();
+ replacement.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void closeForceReleasesOutstandingReservations() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget closing = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ EntryBudget sibling = manager.createEntryBudget(1L, "hive", "file", "file", NONE, NONE);
+ AdmissionReservation leaked = closing.tryReserve(40L).get();
+ AdmissionReservation live = sibling.tryReserve(30L).get();
+
+ closing.close();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ leaked.release();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight(),
+ "a late release from a closed owner must not be subtracted twice");
+ Assertions.assertEquals(30L, sibling.getUsedWeight());
+
+ EntryBudget reopened = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation reReserved = reopened.tryReserve(70L).get();
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ reReserved.release();
+ live.release();
+ reopened.close();
+ sibling.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void peerReclaimCoalescesQueuedRequestsToTheLargestAdmission() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget owner = manager.createEntryBudget(1L, "iceberg", "partition", "partition", NONE, NONE);
+ EntryBudget requester = manager.createEntryBudget(2L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation reservation = owner.tryReserve(100L).get();
+ CountDownLatch firstReclaimStarted = new CountDownLatch(1);
+ CountDownLatch releaseFirstReclaim = new CountDownLatch(1);
+ CountDownLatch secondReclaimFinished = new CountDownLatch(1);
+ AtomicInteger invocations = new AtomicInteger();
+ List<Long> targets = Collections.synchronizedList(new ArrayList<>());
+ owner.setReclaimer(target -> {
+ targets.add(target);
+ if (invocations.getAndIncrement() == 0) {
+ firstReclaimStarted.countDown();
+ await(releaseFirstReclaim);
+ } else {
+ secondReclaimFinished.countDown();
+ }
+ return 0L;
+ });
+ try {
+ requester.requestPeerReclaim(10L);
+ Assertions.assertTrue(firstReclaimStarted.await(10L, TimeUnit.SECONDS));
+
+ requester.requestPeerReclaim(10L);
+ requester.requestPeerReclaim(20L);
+ requester.requestPeerReclaim(15L);
+ releaseFirstReclaim.countDown();
+
+ Assertions.assertTrue(secondReclaimFinished.await(10L, TimeUnit.SECONDS));
+ Assertions.assertEquals(Arrays.asList(10L, 20L), new ArrayList<>(targets),
+ "misses queued behind a running reclamation collapse into one request for the largest");
+ } finally {
+ releaseFirstReclaim.countDown();
+ reservation.release();
+ owner.close();
+ requester.close();
+ }
+ }
+
+ @Test
+ void catalogDeficitNeverAsksAnotherCatalog() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(200L));
+ EntryBudget sibling = manager.createEntryBudget(
+ 1L, "iceberg", "partition", "partition", OptionalLong.of(100L), NONE);
+ EntryBudget requester = manager.createEntryBudget(
+ 1L, "iceberg", "manifest", "manifest", OptionalLong.of(100L), NONE);
+ EntryBudget otherCatalog = manager.createEntryBudget(
+ 2L, "paimon", "partition_view", "partition_view", OptionalLong.of(100L), NONE);
+ AdmissionReservation siblingReservation = sibling.tryReserve(100L).get();
+ // 150 of the global 200 is used, so the request below is short only at the catalog level.
+ AdmissionReservation otherReservation = otherCatalog.tryReserve(50L).get();
+ CountDownLatch siblingAskedTwice = new CountDownLatch(2);
+ List<Long> siblingTargets = Collections.synchronizedList(new ArrayList<>());
+ AtomicInteger otherCatalogAsks = new AtomicInteger();
+ sibling.setReclaimer(target -> {
+ siblingTargets.add(target);
+ siblingAskedTwice.countDown();
+ return 0L;
+ });
+ otherCatalog.setReclaimer(target -> {
+ otherCatalogAsks.incrementAndGet();
+ return 0L;
+ });
+ try {
+ requester.requestPeerReclaim(20L);
+ awaitCount(siblingTargets, 1);
+ // The reclaim executor runs one drain at a time, so the second request is served only after
+ // the first drain has walked every candidate.
+ requester.requestPeerReclaim(20L);
+
+ Assertions.assertTrue(siblingAskedTwice.await(10L, TimeUnit.SECONDS));
+ Assertions.assertEquals(Long.valueOf(20L), siblingTargets.get(0));
+ Assertions.assertEquals(0, otherCatalogAsks.get(),
+ "only this catalog's caches can relieve a catalog-level deficit");
+ } finally {
+ siblingReservation.release();
+ otherReservation.release();
+ sibling.close();
+ requester.close();
+ otherCatalog.close();
+ }
+ }
+
+ @Test
+ void groupDeficitOnlyAsksPhysicalCachesOfTheSameGroup() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(NONE);
+ EntryBudget groupPeer = manager.createEntryBudget(
+ 1L, "iceberg", "mvcc-partition-view", "partition_view", NONE, OptionalLong.of(100L));
+ EntryBudget requester = manager.createEntryBudget(
+ 1L, "iceberg", "list-partitions-view", "partition_view", NONE, OptionalLong.of(100L));
+ EntryBudget otherGroup = manager.createEntryBudget(
+ 1L, "iceberg", "manifest", "manifest", NONE, OptionalLong.of(500L));
+ AdmissionReservation peerReservation = groupPeer.tryReserve(100L).get();
+ AdmissionReservation otherReservation = otherGroup.tryReserve(400L).get();
+ CountDownLatch peerAskedTwice = new CountDownLatch(2);
+ List<Long> peerTargets = Collections.synchronizedList(new ArrayList<>());
+ AtomicInteger otherGroupAsks = new AtomicInteger();
+ groupPeer.setReclaimer(target -> {
+ peerTargets.add(target);
+ peerAskedTwice.countDown();
+ return 0L;
+ });
+ otherGroup.setReclaimer(target -> {
+ otherGroupAsks.incrementAndGet();
+ return 0L;
+ });
+ try {
+ requester.requestPeerReclaim(20L);
+ awaitCount(peerTargets, 1);
+ requester.requestPeerReclaim(20L);
+
+ Assertions.assertTrue(peerAskedTwice.await(10L, TimeUnit.SECONDS));
+ Assertions.assertEquals(Long.valueOf(20L), peerTargets.get(0));
+ Assertions.assertEquals(0, otherGroupAsks.get(),
+ "the larger cache of another group can not relieve this group's shared limit");
+ } finally {
+ peerReservation.release();
+ otherReservation.release();
+ groupPeer.close();
+ requester.close();
+ otherGroup.close();
+ }
+ }
+
+ @Test
+ void catalogAndGroupBucketsLiveExactlyAsLongAsTheirEntries() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(1L << 20));
+ List<EntryBudget> budgets = new ArrayList<>();
+ for (long catalogId = 1L; catalogId <= 3L; catalogId++) {
+ for (String entry : new String[] {"paimon-table", "partition_view"}) {
Review Comment:
[P2] Cover retirement of a shared entry group. This loop gives each physical budget a different group, so the group-map assertions cannot catch removing a `partition_view` bucket while its other physical cache is still live. If that bucket is removed on the first close, a newly created cache gets a fresh limit and can oversubscribe the still-live sibling's reservation. Add two budgets with the same group, close one, recreate it, and assert the shared limit until the last sibling closes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6082420666
<details>
<summary>TPC-H: <b>Total hot run time: 28639 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d76b69ce222e25f81491146f11209846de0840b, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17604 5659 5534 5534
q2 2154 305 276 276
q3 10186 1397 832 832
q4 4755 479 345 345
q5 7440 786 536 536
q6 186 192 153 153
q7 758 790 607 607
q8 9259 1331 1025 1025
q9 5984 4438 4421 4421
q10 6824 1294 1022 1022
q11 418 255 227 227
q12 626 410 290 290
q13 18060 3063 2379 2379
q14 270 273 242 242
q15 q16 733 728 665 665
q17 1216 744 592 592
q18 7141 6259 6188 6188
q19 1108 959 596 596
q20 379 332 221 221
q21 5055 2221 2440 2221
q22 356 314 267 267
Total cold run time: 100512 ms
Total hot run time: 28639 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6174 6135 6125 6125
q2 694 565 555 555
q3 5219 5434 4942 4942
q4 2024 2114 1538 1538
q5 5290 5144 6130 5144
q6 250 200 153 153
q7 2354 2140 1914 1914
q8 3271 3063 2762 2762
q9 8326 8285 8328 8285
q10 3787 3724 3337 3337
q11 635 429 405 405
q12 687 717 519 519
q13 2927 3204 2485 2485
q14 340 331 292 292
q15 q16 704 710 700 700
q17 8484 7526 7365 7365
q18 13211 12325 13157 12325
q19 982 826 826 826
q20 2249 2193 2002 2002
q21 5924 4668 4759 4668
q22 528 446 393 393
Total cold run time: 74060 ms
Total hot run time: 66735 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs merged PR #68839:
URL: https://github.com/apache/doris/pull/68839
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68839:
URL: https://github.com/apache/doris/pull/68839#discussion_r4235999737
##########
fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheBudgetManagerTest.java:
##########
@@ -0,0 +1,464 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.cache;
+
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.AdmissionReservation;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.EntryBudget;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.ReservationReplacement;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Budget arithmetic of {@link MetaCacheBudgetManager} without a cache on top: the global, catalog and
+ * entry-group limits, the reservation hand-off between two generations of one key, release on close,
+ * and which peers a reclamation request may ask.
+ */
+class MetaCacheBudgetManagerTest {
+ private static final OptionalLong NONE = OptionalLong.empty();
+
+ @Test
+ void reservationsRespectGlobalCatalogAndEntryLimits() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget file = manager.createEntryBudget(
+ 1L, "hive", "file", "file", OptionalLong.of(80L), OptionalLong.of(60L));
+ EntryBudget partitionView = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", OptionalLong.of(80L), OptionalLong.of(50L));
+
+ AdmissionReservation fileReservation = file.tryReserve(60L).get();
+ Assertions.assertFalse(partitionView.tryReserve(30L).isPresent(),
+ "the catalog limit of 80 is already 60 used");
+ AdmissionReservation viewReservation = partitionView.tryReserve(20L).get();
+ Assertions.assertEquals(80L, manager.getGlobalUsedWeight());
+ Assertions.assertFalse(partitionView.tryReplace(viewReservation, 30L).isPresent());
+
+ fileReservation.release();
+ ReservationReplacement grown = partitionView.tryReplace(viewReservation, 30L).get();
+ grown.commit();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ Assertions.assertEquals(30L, partitionView.getUsedWeight());
+
+ grown.current().release();
+ Assertions.assertFalse(partitionView.tryReserve(51L).isPresent(),
+ "the entry limit of 50 applies even when the catalog and global budgets have room");
+ file.close();
+ partitionView.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void concurrentReservationsNeverExceedTheGlobalLimit() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "manifest", "manifest", NONE, NONE);
+ ExecutorService executor = Executors.newFixedThreadPool(8);
+ CountDownLatch start = new CountDownLatch(1);
+ List<AdmissionReservation> reservations = Collections.synchronizedList(new ArrayList<>());
+ List<Future<?>> attempts = new ArrayList<>();
+ try {
+ for (int i = 0; i < 200; i++) {
+ attempts.add(executor.submit(() -> {
+ await(start);
+ budget.tryReserve(1L).ifPresent(reservations::add);
+ }));
+ }
+ start.countDown();
+ executor.shutdown();
+ Assertions.assertTrue(executor.awaitTermination(10L, TimeUnit.SECONDS));
+ // A refused attempt returns an empty Optional; an attempt that threw would surface here.
+ for (Future<?> attempt : attempts) {
+ attempt.get();
+ }
+ Assertions.assertEquals(100, reservations.size());
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ } finally {
+ executor.shutdownNow();
+ reservations.forEach(AdmissionReservation::release);
+ budget.close();
+ }
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void entryBudgetIsClampedToTheSmallestEnclosingLimit() {
+ // A catalog created on an FE with a larger global quota replays here with limits above ours.
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget replayed = manager.createEntryBudget(
+ 1L, "iceberg", "table", "table", OptionalLong.of(400L), OptionalLong.of(300L));
+ Assertions.assertEquals(100L, replayed.getEffectiveMaxWeight());
+ AdmissionReservation reservation = replayed.tryReserve(100L).get();
+ Assertions.assertFalse(replayed.tryReserve(1L).isPresent());
+ reservation.release();
+ replayed.close();
+
+ EntryBudget entryAboveCatalog = manager.createEntryBudget(
+ 2L, "iceberg", "table", "table", OptionalLong.of(40L), OptionalLong.of(60L));
+ Assertions.assertEquals(40L, entryAboveCatalog.getEffectiveMaxWeight());
+ Assertions.assertFalse(entryAboveCatalog.tryReserve(41L).isPresent());
+ entryAboveCatalog.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void weightParsingAcceptsBinaryUnitsAndHeapPercentages() {
+ long heap = 4L * 1024L * 1024L * 1024L;
+ String key = "external_meta_cache_max_weight";
+ Assertions.assertEquals(1L, CacheSpec.parseWeight("1", key, true, heap));
+ Assertions.assertEquals(2L * 1024L * 1024L, CacheSpec.parseWeight(" 2 mb ", key, true, heap));
+ Assertions.assertEquals(heap / 4L, CacheSpec.parseWeight("25%", key, true, heap));
+ Assertions.assertEquals(heap / 8L, CacheSpec.parseWeight("12.5%", key, true, heap));
+ // 0 disables the global quota; rejecting 0% is the caller's job (ExternalMetaCacheMgr).
+ Assertions.assertEquals(0L, CacheSpec.parseWeight("0", key, true, heap));
+ Assertions.assertEquals(0L, CacheSpec.parseWeight("0%", key, true, heap));
+
+ for (String invalid : new String[] {"", "1.5GB", "101%", "-1%", "-1", "8EB", "1PB000",
+ "8192PB", "999999999999999999PB"}) {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> CacheSpec.parseWeight(invalid, key, true, heap), invalid);
+ }
+ IllegalArgumentException catalogPercentage = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> CacheSpec.parseWeight("10%", "meta.cache.max-weight", false, heap));
+ Assertions.assertEquals("Invalid cache weight for 'meta.cache.max-weight': 10%",
+ catalogPercentage.getMessage());
+ }
+
+ @Test
+ void releasingAReservationTwiceReleasesItOnce() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget first = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ EntryBudget second = manager.createEntryBudget(1L, "hive", "file", "file", NONE, NONE);
+ AdmissionReservation released = first.tryReserve(40L).get();
+ AdmissionReservation kept = second.tryReserve(30L).get();
+
+ released.release();
+ released.release();
+
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight(),
+ "a repeated release must not take bytes from another reservation");
+ Assertions.assertEquals(0L, released.getBytes());
+ kept.release();
+ first.close();
+ second.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void replacementHoldsTheLargerGenerationUntilCommitOrRollback() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "table", "table", NONE, NONE);
+ AdmissionReservation first = budget.tryReserve(30L).get();
+
+ ReservationReplacement grow = budget.tryReplace(first, 50L).get();
+ Assertions.assertEquals(50L, manager.getGlobalUsedWeight(), "max(old, new) is held, never old + new");
+ Assertions.assertFalse(budget.tryReplace(first, 10L).isPresent(),
+ "a generation that is being replaced can not be replaced again");
+ grow.commit();
+ AdmissionReservation second = grow.current();
+ Assertions.assertEquals(50L, second.getBytes());
+
+ ReservationReplacement shrink = budget.tryReplace(second, 10L).get();
+ Assertions.assertEquals(50L, manager.getGlobalUsedWeight(),
+ "the old generation stays charged until the new one is published");
+ shrink.commit();
+ Assertions.assertEquals(10L, manager.getGlobalUsedWeight());
+
+ AdmissionReservation third = shrink.current();
+ ReservationReplacement abandoned = budget.tryReplace(third, 40L).get();
+ Assertions.assertEquals(40L, manager.getGlobalUsedWeight());
+ abandoned.rollback();
+ Assertions.assertEquals(10L, manager.getGlobalUsedWeight());
+ Assertions.assertThrows(IllegalStateException.class, abandoned::commit);
+ third.release();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight(), "rollback reactivates the previous generation");
+ budget.close();
+ }
+
+ @Test
+ void closedBudgetRejectsNewReservationsAndReplacements() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget stale = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation zeroBytes = stale.tryReserve(0L).get();
+
+ stale.close();
+ stale.close();
+
+ Assertions.assertFalse(stale.tryReserve(1L).isPresent());
+ Assertions.assertFalse(stale.tryReplace(zeroBytes, 1L).isPresent());
+ zeroBytes.release();
+
+ EntryBudget replacement = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation full = replacement.tryReserve(100L).get();
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ full.release();
+ replacement.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void closeForceReleasesOutstandingReservations() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget closing = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ EntryBudget sibling = manager.createEntryBudget(1L, "hive", "file", "file", NONE, NONE);
+ AdmissionReservation leaked = closing.tryReserve(40L).get();
+ AdmissionReservation live = sibling.tryReserve(30L).get();
+
+ closing.close();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ leaked.release();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight(),
+ "a late release from a closed owner must not be subtracted twice");
+ Assertions.assertEquals(30L, sibling.getUsedWeight());
+
+ EntryBudget reopened = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation reReserved = reopened.tryReserve(70L).get();
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ reReserved.release();
+ live.release();
+ reopened.close();
+ sibling.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void peerReclaimCoalescesQueuedRequestsToTheLargestAdmission() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget owner = manager.createEntryBudget(1L, "iceberg", "partition", "partition", NONE, NONE);
+ EntryBudget requester = manager.createEntryBudget(2L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation reservation = owner.tryReserve(100L).get();
+ CountDownLatch firstReclaimStarted = new CountDownLatch(1);
+ CountDownLatch releaseFirstReclaim = new CountDownLatch(1);
+ CountDownLatch secondReclaimFinished = new CountDownLatch(1);
+ AtomicInteger invocations = new AtomicInteger();
+ List<Long> targets = Collections.synchronizedList(new ArrayList<>());
+ owner.setReclaimer(target -> {
+ targets.add(target);
+ if (invocations.getAndIncrement() == 0) {
+ firstReclaimStarted.countDown();
+ await(releaseFirstReclaim);
+ } else {
+ secondReclaimFinished.countDown();
+ }
+ return 0L;
+ });
+ try {
+ requester.requestPeerReclaim(10L);
+ Assertions.assertTrue(firstReclaimStarted.await(10L, TimeUnit.SECONDS));
+
+ requester.requestPeerReclaim(10L);
+ requester.requestPeerReclaim(20L);
+ requester.requestPeerReclaim(15L);
+ releaseFirstReclaim.countDown();
+
+ Assertions.assertTrue(secondReclaimFinished.await(10L, TimeUnit.SECONDS));
+ Assertions.assertEquals(Arrays.asList(10L, 20L), new ArrayList<>(targets),
Review Comment:
Fixed in 3506202c47e for the coalescing test. Before it checks the targets, it now waits on two barriers. Each barrier is a drain of a separate budget manager on the same process-wide reclaim thread. The first runs after the second drain has finished and queued anything it left behind; the second runs after that queued drain. With a mutation that leaves the request pending and pauses 50 ms at the start of each drain, the previous version of the test passes and this one fails with `[10, 20, 20, 20]`.
The catalog and group exclusion tests stay as they are. What they guard is the first drain's walk over the candidates, and that walk is already complete when their latch fires, because the second request is served only after the first drain returns. A regression that asked the other catalog or the other group would do so in that first drain (in the group test the other group's larger cache even sorts first), so the counter is already non-zero when the assertion runs. The unobserved tail of the second drain only repeats the same walk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6086790372
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6087751854
<details>
<summary>TPC-DS: <b>Total hot run time: 151332 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 742c58b19b104382a449f16bac1635a01b8e063c, data reload: false
query5 4314 582 446 446
query6 417 206 190 190
query7 4804 442 226 226
query8 315 183 163 163
query9 8795 3916 3943 3916
query10 477 319 259 259
query11 5981 3450 3196 3196
query12 147 90 85 85
query13 1267 436 317 317
query14 6530 4736 4486 4486
query14_1 4203 4218 4216 4216
query15 203 198 179 179
query16 940 436 410 410
query17 860 667 533 533
query18 2413 419 308 308
query19 193 160 140 140
query20 81 78 78 78
query21 205 130 115 115
query22 12846 12852 12737 12737
query23 13116 12577 12088 12088
query23_1 12233 12217 12145 12145
query24 7094 847 461 461
query24_1 478 473 490 473
query25 540 423 368 368
query26 1310 263 137 137
query27 2770 436 269 269
query28 4610 1924 1918 1918
query29 1592 586 462 462
query30 298 211 180 180
query31 878 765 657 657
query32 140 92 92 92
query33 509 301 249 249
query34 912 840 494 494
query35 748 766 634 634
query36 796 807 751 751
query37 131 109 91 91
query38 1789 1751 1678 1678
query39 719 704 670 670
query39_1 673 677 682 677
query40 220 127 99 99
query41 73 73 72 72
query42 90 82 82 82
query43 361 379 331 331
query44 1288 692 691 691
query45 180 183 161 161
query46 847 969 579 579
query47 2991 2999 2839 2839
query48 312 351 209 209
query49 561 380 292 292
query50 660 266 207 207
query51 10519 10516 10502 10502
query52 80 77 70 70
query53 189 210 146 146
query54 250 208 182 182
query55 70 67 68 67
query56 220 202 214 202
query57 1644 1586 1532 1532
query58 275 255 259 255
query59 2348 2364 2121 2121
query60 285 235 212 212
query61 143 147 142 142
query62 399 340 287 287
query63 193 156 160 156
query64 2715 972 794 794
query65 3453 3367 3358 3358
query66 1783 402 299 299
query67 20064 19956 20239 19956
query68 2986 1016 625 625
query69 401 295 250 250
query70 936 835 889 835
query71 289 237 201 201
query72 2663 2576 2297 2297
query73 524 517 302 302
query74 4572 4461 4279 4279
query75 2345 2310 1939 1939
query76 2286 1017 586 586
query77 352 386 294 294
query78 9012 8943 8397 8397
query79 982 830 504 504
query80 1229 414 332 332
query81 583 326 265 265
query82 590 134 113 113
query83 291 195 176 176
query84 313 128 97 97
query85 895 415 362 362
query86 400 239 226 226
query87 2010 1925 1815 1815
query88 3588 2684 2652 2652
query89 337 310 254 254
query90 1884 179 169 169
query91 158 145 124 124
query92 95 89 90 89
query93 1091 966 554 554
query94 636 296 277 277
query95 581 350 376 350
query96 660 531 229 229
query97 2454 2411 2307 2307
query98 155 162 141 141
query99 776 755 650 650
Total cold run time: 234386 ms
Total hot run time: 151332 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6094332367
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68839_3506202c47e3aa4ad188b12227c64afa8f83f029/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68839_3506202c47e3aa4ad188b12227c64afa8f83f029/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6093281467
<details>
<summary>ClickBench: <b>Total hot run time: 24.52 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3506202c47e3aa4ad188b12227c64afa8f83f029, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.12 0.12
query4 1.62 0.13 0.12
query5 0.28 0.26 0.26
query6 1.17 0.69 0.68
query7 0.04 0.02 0.01
query8 0.05 0.04 0.04
query9 0.44 0.36 0.36
query10 0.58 0.62 0.59
query11 0.20 0.14 0.14
query12 0.19 0.15 0.14
query13 0.51 0.49 0.50
query14 0.98 0.96 0.95
query15 0.63 0.60 0.61
query16 0.34 0.31 0.32
query17 1.17 1.13 1.15
query18 0.22 0.21 0.21
query19 2.17 2.01 1.99
query20 0.02 0.01 0.01
query21 15.42 0.27 0.14
query22 4.47 0.06 0.05
query23 16.28 0.33 0.12
query24 3.25 0.56 0.35
query25 0.12 0.06 0.05
query26 1.01 0.24 0.15
query27 0.04 0.04 0.04
query28 1.75 0.63 0.41
query29 12.57 4.13 3.27
query30 0.29 0.15 0.14
query31 2.77 0.58 0.34
query32 3.22 0.62 0.52
query33 3.14 3.22 3.22
query34 15.59 4.02 3.38
query35 3.34 3.34 3.36
query36 0.57 0.43 0.46
query37 0.10 0.06 0.07
query38 0.04 0.04 0.04
query39 0.04 0.03 0.03
query40 0.19 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95.35 s
Total hot run time: 24.52 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6081237488
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6081238639
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6086789895
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68839:
URL: https://github.com/apache/doris/pull/68839#discussion_r4233435702
##########
fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheBudgetManagerTest.java:
##########
@@ -0,0 +1,464 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.cache;
+
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.AdmissionReservation;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.EntryBudget;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.ReservationReplacement;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Budget arithmetic of {@link MetaCacheBudgetManager} without a cache on top: the global, catalog and
+ * entry-group limits, the reservation hand-off between two generations of one key, release on close,
+ * and which peers a reclamation request may ask.
+ */
+class MetaCacheBudgetManagerTest {
+ private static final OptionalLong NONE = OptionalLong.empty();
+
+ @Test
+ void reservationsRespectGlobalCatalogAndEntryLimits() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget file = manager.createEntryBudget(
+ 1L, "hive", "file", "file", OptionalLong.of(80L), OptionalLong.of(60L));
+ EntryBudget partitionView = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", OptionalLong.of(80L), OptionalLong.of(50L));
+
+ AdmissionReservation fileReservation = file.tryReserve(60L).get();
+ Assertions.assertFalse(partitionView.tryReserve(30L).isPresent(),
+ "the catalog limit of 80 is already 60 used");
+ AdmissionReservation viewReservation = partitionView.tryReserve(20L).get();
+ Assertions.assertEquals(80L, manager.getGlobalUsedWeight());
+ Assertions.assertFalse(partitionView.tryReplace(viewReservation, 30L).isPresent());
+
+ fileReservation.release();
+ ReservationReplacement grown = partitionView.tryReplace(viewReservation, 30L).get();
+ grown.commit();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ Assertions.assertEquals(30L, partitionView.getUsedWeight());
+
+ grown.current().release();
+ Assertions.assertFalse(partitionView.tryReserve(51L).isPresent(),
+ "the entry limit of 50 applies even when the catalog and global budgets have room");
+ file.close();
+ partitionView.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void concurrentReservationsNeverExceedTheGlobalLimit() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "manifest", "manifest", NONE, NONE);
+ ExecutorService executor = Executors.newFixedThreadPool(8);
+ CountDownLatch start = new CountDownLatch(1);
+ List<AdmissionReservation> reservations = Collections.synchronizedList(new ArrayList<>());
+ List<Future<?>> attempts = new ArrayList<>();
+ try {
+ for (int i = 0; i < 200; i++) {
+ attempts.add(executor.submit(() -> {
+ await(start);
+ budget.tryReserve(1L).ifPresent(reservations::add);
+ }));
+ }
+ start.countDown();
+ executor.shutdown();
+ Assertions.assertTrue(executor.awaitTermination(10L, TimeUnit.SECONDS));
+ // A refused attempt returns an empty Optional; an attempt that threw would surface here.
+ for (Future<?> attempt : attempts) {
+ attempt.get();
+ }
+ Assertions.assertEquals(100, reservations.size());
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ } finally {
+ executor.shutdownNow();
+ reservations.forEach(AdmissionReservation::release);
+ budget.close();
+ }
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void entryBudgetIsClampedToTheSmallestEnclosingLimit() {
+ // A catalog created on an FE with a larger global quota replays here with limits above ours.
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget replayed = manager.createEntryBudget(
+ 1L, "iceberg", "table", "table", OptionalLong.of(400L), OptionalLong.of(300L));
+ Assertions.assertEquals(100L, replayed.getEffectiveMaxWeight());
+ AdmissionReservation reservation = replayed.tryReserve(100L).get();
+ Assertions.assertFalse(replayed.tryReserve(1L).isPresent());
+ reservation.release();
+ replayed.close();
+
+ EntryBudget entryAboveCatalog = manager.createEntryBudget(
+ 2L, "iceberg", "table", "table", OptionalLong.of(40L), OptionalLong.of(60L));
+ Assertions.assertEquals(40L, entryAboveCatalog.getEffectiveMaxWeight());
+ Assertions.assertFalse(entryAboveCatalog.tryReserve(41L).isPresent());
+ entryAboveCatalog.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void weightParsingAcceptsBinaryUnitsAndHeapPercentages() {
+ long heap = 4L * 1024L * 1024L * 1024L;
+ String key = "external_meta_cache_max_weight";
+ Assertions.assertEquals(1L, CacheSpec.parseWeight("1", key, true, heap));
+ Assertions.assertEquals(2L * 1024L * 1024L, CacheSpec.parseWeight(" 2 mb ", key, true, heap));
+ Assertions.assertEquals(heap / 4L, CacheSpec.parseWeight("25%", key, true, heap));
+ Assertions.assertEquals(heap / 8L, CacheSpec.parseWeight("12.5%", key, true, heap));
+ // 0 disables the global quota; rejecting 0% is the caller's job (ExternalMetaCacheMgr).
+ Assertions.assertEquals(0L, CacheSpec.parseWeight("0", key, true, heap));
+ Assertions.assertEquals(0L, CacheSpec.parseWeight("0%", key, true, heap));
+
+ for (String invalid : new String[] {"", "1.5GB", "101%", "-1%", "-1", "8EB", "1PB000",
+ "8192PB", "999999999999999999PB"}) {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> CacheSpec.parseWeight(invalid, key, true, heap), invalid);
+ }
+ IllegalArgumentException catalogPercentage = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> CacheSpec.parseWeight("10%", "meta.cache.max-weight", false, heap));
+ Assertions.assertEquals("Invalid cache weight for 'meta.cache.max-weight': 10%",
+ catalogPercentage.getMessage());
+ }
+
+ @Test
+ void releasingAReservationTwiceReleasesItOnce() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget first = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ EntryBudget second = manager.createEntryBudget(1L, "hive", "file", "file", NONE, NONE);
+ AdmissionReservation released = first.tryReserve(40L).get();
+ AdmissionReservation kept = second.tryReserve(30L).get();
+
+ released.release();
+ released.release();
+
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight(),
+ "a repeated release must not take bytes from another reservation");
+ Assertions.assertEquals(0L, released.getBytes());
+ kept.release();
+ first.close();
+ second.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void replacementHoldsTheLargerGenerationUntilCommitOrRollback() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "table", "table", NONE, NONE);
+ AdmissionReservation first = budget.tryReserve(30L).get();
+
+ ReservationReplacement grow = budget.tryReplace(first, 50L).get();
+ Assertions.assertEquals(50L, manager.getGlobalUsedWeight(), "max(old, new) is held, never old + new");
+ Assertions.assertFalse(budget.tryReplace(first, 10L).isPresent(),
+ "a generation that is being replaced can not be replaced again");
+ grow.commit();
+ AdmissionReservation second = grow.current();
+ Assertions.assertEquals(50L, second.getBytes());
+
+ ReservationReplacement shrink = budget.tryReplace(second, 10L).get();
+ Assertions.assertEquals(50L, manager.getGlobalUsedWeight(),
+ "the old generation stays charged until the new one is published");
+ shrink.commit();
+ Assertions.assertEquals(10L, manager.getGlobalUsedWeight());
+
+ AdmissionReservation third = shrink.current();
+ ReservationReplacement abandoned = budget.tryReplace(third, 40L).get();
+ Assertions.assertEquals(40L, manager.getGlobalUsedWeight());
+ abandoned.rollback();
+ Assertions.assertEquals(10L, manager.getGlobalUsedWeight());
+ Assertions.assertThrows(IllegalStateException.class, abandoned::commit);
+ third.release();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight(), "rollback reactivates the previous generation");
+ budget.close();
+ }
+
+ @Test
+ void closedBudgetRejectsNewReservationsAndReplacements() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget stale = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation zeroBytes = stale.tryReserve(0L).get();
+
+ stale.close();
+ stale.close();
+
+ Assertions.assertFalse(stale.tryReserve(1L).isPresent());
+ Assertions.assertFalse(stale.tryReplace(zeroBytes, 1L).isPresent());
+ zeroBytes.release();
+
+ EntryBudget replacement = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation full = replacement.tryReserve(100L).get();
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ full.release();
+ replacement.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void closeForceReleasesOutstandingReservations() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget closing = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ EntryBudget sibling = manager.createEntryBudget(1L, "hive", "file", "file", NONE, NONE);
+ AdmissionReservation leaked = closing.tryReserve(40L).get();
+ AdmissionReservation live = sibling.tryReserve(30L).get();
+
+ closing.close();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ leaked.release();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight(),
+ "a late release from a closed owner must not be subtracted twice");
+ Assertions.assertEquals(30L, sibling.getUsedWeight());
+
+ EntryBudget reopened = manager.createEntryBudget(1L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation reReserved = reopened.tryReserve(70L).get();
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ reReserved.release();
+ live.release();
+ reopened.close();
+ sibling.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void peerReclaimCoalescesQueuedRequestsToTheLargestAdmission() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget owner = manager.createEntryBudget(1L, "iceberg", "partition", "partition", NONE, NONE);
+ EntryBudget requester = manager.createEntryBudget(2L, "hive", "partition_view", "partition_view", NONE, NONE);
+ AdmissionReservation reservation = owner.tryReserve(100L).get();
+ CountDownLatch firstReclaimStarted = new CountDownLatch(1);
+ CountDownLatch releaseFirstReclaim = new CountDownLatch(1);
+ CountDownLatch secondReclaimFinished = new CountDownLatch(1);
+ AtomicInteger invocations = new AtomicInteger();
+ List<Long> targets = Collections.synchronizedList(new ArrayList<>());
+ owner.setReclaimer(target -> {
+ targets.add(target);
+ if (invocations.getAndIncrement() == 0) {
+ firstReclaimStarted.countDown();
+ await(releaseFirstReclaim);
+ } else {
+ secondReclaimFinished.countDown();
+ }
+ return 0L;
+ });
+ try {
+ requester.requestPeerReclaim(10L);
+ Assertions.assertTrue(firstReclaimStarted.await(10L, TimeUnit.SECONDS));
+
+ requester.requestPeerReclaim(10L);
+ requester.requestPeerReclaim(20L);
+ requester.requestPeerReclaim(15L);
+ releaseFirstReclaim.countDown();
+
+ Assertions.assertTrue(secondReclaimFinished.await(10L, TimeUnit.SECONDS));
+ Assertions.assertEquals(Arrays.asList(10L, 20L), new ArrayList<>(targets),
Review Comment:
[P2] Wait for the reclaim drain to finish before asserting coalescing. `secondReclaimFinished` is counted down inside the second reclaimer, before that callback returns and before `drainPeerReclaim` decides whether to queue another drain. If coalescing regresses and leaves a request pending, this assertion can see `[10, 20]` and pass before a third callback runs; the `finally` cleanup can also close the requester before that work is observed. Add a deterministic drain-completion barrier or test hook, then assert the final callback list. The catalog/group exclusion checks have the same end-of-second-drain observation gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68839:
URL: https://github.com/apache/doris/pull/68839#discussion_r4236050867
##########
fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonMetaCacheCatalogTest.java:
##########
@@ -422,6 +422,39 @@ void realFileStoreTableIsAdmittedByTheWeightGovernedCache(@TempDir java.nio.file
Assertions.assertEquals(0L, budgetManager.getGlobalUsedWeight());
}
+ @Test
+ void tableInvalidationReleasesTheReservationAndTheReloadIsReadmitted(@TempDir java.nio.file.Path warehouse)
+ throws Exception {
+ LocalFileIO fileIO = LocalFileIO.create();
+ org.apache.paimon.fs.Path tablePath =
+ createFileStoreTable(fileIO, warehouse.resolve("refreshed-table"), "payload").location();
+ RecordingCatalog recording = new RecordingCatalog();
+ recording.tableSupplier = () -> FileStoreTableFactory.create(fileIO, tablePath);
+ MetaCacheBudgetManager budgetManager = new MetaCacheBudgetManager(OptionalLong.of(1024L * 1024L));
+ try (CatalogMetaCache owner = new CatalogMetaCache(
+ budgetManager, 66717L, "paimon", Collections.emptyMap())) {
+ PaimonMetaCacheCatalog catalog = new PaimonMetaCacheCatalog(recording.catalog(), owner,
+ 100, 100, cacheOptions(Duration.ofDays(1), Duration.ofDays(1)),
+ true, System::nanoTime);
+
+ Table first = catalog.getTable(TABLE);
+ long admittedWeight = budgetManager.getGlobalUsedWeight();
+ Assertions.assertTrue(admittedWeight > 0L);
+
+ // REFRESH TABLE reaches the connector as a table-scoped invalidation of its cache owner.
+ owner.invalidateTable("db", "t");
Review Comment:
[P2] Exercise the connector hook in this invalidation test. Calling `owner.invalidateTable` here bypasses the `PaimonConnector.invalidateTable` path used by REFRESH TABLE. If that hook stops forwarding table invalidation to the shared owner, this test still releases the reservation and reloads successfully while production continues to serve a stale weighted `paimon-table` entry. Load the table through a connector-owned catalog, call `connector.invalidateTable("db", "t")`, and assert that its reservation is released and the next read reloads.
##########
fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheBudgetManagerTest.java:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.cache;
+
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.AdmissionReservation;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.EntryBudget;
+import org.apache.doris.connector.cache.MetaCacheBudgetManager.ReservationReplacement;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Budget arithmetic of {@link MetaCacheBudgetManager} without a cache on top: the global, catalog and
+ * entry-group limits, the reservation hand-off between two generations of one key, release on close,
+ * and which peers a reclamation request may ask.
+ */
+class MetaCacheBudgetManagerTest {
+ private static final OptionalLong NONE = OptionalLong.empty();
+
+ @Test
+ void reservationsRespectGlobalCatalogAndEntryLimits() {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget file = manager.createEntryBudget(
+ 1L, "hive", "file", "file", OptionalLong.of(80L), OptionalLong.of(60L));
+ EntryBudget partitionView = manager.createEntryBudget(
+ 1L, "hive", "partition_view", "partition_view", OptionalLong.of(80L), OptionalLong.of(50L));
+
+ AdmissionReservation fileReservation = file.tryReserve(60L).get();
+ Assertions.assertFalse(partitionView.tryReserve(30L).isPresent(),
+ "the catalog limit of 80 is already 60 used");
+ AdmissionReservation viewReservation = partitionView.tryReserve(20L).get();
+ Assertions.assertEquals(80L, manager.getGlobalUsedWeight());
+ Assertions.assertFalse(partitionView.tryReplace(viewReservation, 30L).isPresent());
+
+ fileReservation.release();
+ ReservationReplacement grown = partitionView.tryReplace(viewReservation, 30L).get();
+ grown.commit();
+ Assertions.assertEquals(30L, manager.getGlobalUsedWeight());
+ Assertions.assertEquals(30L, partitionView.getUsedWeight());
+
+ grown.current().release();
+ Assertions.assertFalse(partitionView.tryReserve(51L).isPresent(),
+ "the entry limit of 50 applies even when the catalog and global budgets have room");
+ file.close();
+ partitionView.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void concurrentReservationsNeverExceedTheGlobalLimit() throws Exception {
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget budget = manager.createEntryBudget(1L, "iceberg", "manifest", "manifest", NONE, NONE);
+ ExecutorService executor = Executors.newFixedThreadPool(8);
+ CountDownLatch start = new CountDownLatch(1);
+ List<AdmissionReservation> reservations = Collections.synchronizedList(new ArrayList<>());
+ List<Future<?>> attempts = new ArrayList<>();
+ try {
+ for (int i = 0; i < 200; i++) {
+ attempts.add(executor.submit(() -> {
+ await(start);
+ budget.tryReserve(1L).ifPresent(reservations::add);
+ }));
+ }
+ start.countDown();
+ executor.shutdown();
+ Assertions.assertTrue(executor.awaitTermination(10L, TimeUnit.SECONDS));
+ // A refused attempt returns an empty Optional; an attempt that threw would surface here.
+ for (Future<?> attempt : attempts) {
+ attempt.get();
+ }
+ Assertions.assertEquals(100, reservations.size());
+ Assertions.assertEquals(100L, manager.getGlobalUsedWeight());
+ } finally {
+ executor.shutdownNow();
+ reservations.forEach(AdmissionReservation::release);
+ budget.close();
+ }
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void entryBudgetIsClampedToTheSmallestEnclosingLimit() {
+ // A catalog created on an FE with a larger global quota replays here with limits above ours.
+ MetaCacheBudgetManager manager = new MetaCacheBudgetManager(OptionalLong.of(100L));
+ EntryBudget replayed = manager.createEntryBudget(
+ 1L, "iceberg", "table", "table", OptionalLong.of(400L), OptionalLong.of(300L));
+ Assertions.assertEquals(100L, replayed.getEffectiveMaxWeight());
+ AdmissionReservation reservation = replayed.tryReserve(100L).get();
+ Assertions.assertFalse(replayed.tryReserve(1L).isPresent());
+ reservation.release();
+ replayed.close();
+
+ EntryBudget entryAboveCatalog = manager.createEntryBudget(
+ 2L, "iceberg", "table", "table", OptionalLong.of(40L), OptionalLong.of(60L));
+ Assertions.assertEquals(40L, entryAboveCatalog.getEffectiveMaxWeight());
+ Assertions.assertFalse(entryAboveCatalog.tryReserve(41L).isPresent());
+ entryAboveCatalog.close();
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+
+ @Test
+ void weightParsingAcceptsBinaryUnitsAndHeapPercentages() {
+ long heap = 4L * 1024L * 1024L * 1024L;
+ String key = "external_meta_cache_max_weight";
Review Comment:
[P2] Test that the FE global quota reaches managed connector caches. These assertions call `CacheSpec.parseWeight` directly, and the budget tests instantiate their own managers; none exercises `ExternalMetaCacheMgr` passing `Config.external_meta_cache_max_weight` to `MetaCacheGovernance`. If that bridge is removed, an operator's global limit stops bounding connector caches while every new test still passes. Add an FE-side case that configures a small global limit, initializes the cache manager, and verifies combined admission across managed catalogs.
##########
fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java:
##########
@@ -226,6 +229,50 @@ private static Table fakeTable(String name) {
PartitionSpec.unpartitioned(), "s3://b/" + name, Collections.emptyMap());
}
+ @Test
+ public void weightLimitsReachEveryConnectorOwnedCache() throws Exception {
+ long oneMb = 1024L * 1024L;
+ Map<String, Long> catalogLimited = maxWeightsOf(props("meta.cache.max-weight", "1MB"));
+ for (String cache : new String[] {"iceberg-table", "iceberg-latest-snapshot", "iceberg-partition",
+ "iceberg-format", "iceberg-manifest", "iceberg-equality-delete-field-ids",
+ "iceberg.mvcc-partition-view", "iceberg.list-partitions-view"}) {
+ Assertions.assertEquals(Long.valueOf(oneMb), catalogLimited.get(cache),
+ "a catalog limit must bound " + cache);
+ }
+
+ Map<String, Long> manifestLimited = maxWeightsOf(props("meta.cache.iceberg.manifest.max-weight", "1MB"));
+ Assertions.assertEquals(Long.valueOf(oneMb), manifestLimited.get("iceberg-manifest"));
+ Assertions.assertEquals(Long.valueOf(-1L), manifestLimited.get("iceberg-table"),
+ "an entry limit must bound only its own entry");
+
+ Map<String, Long> viewLimited = maxWeightsOf(props("meta.cache.iceberg.partition_view.max-weight", "1MB"));
+ Assertions.assertEquals(Long.valueOf(oneMb), viewLimited.get("iceberg.mvcc-partition-view"));
+ Assertions.assertEquals(Long.valueOf(oneMb), viewLimited.get("iceberg.list-partitions-view"),
Review Comment:
[P2] Verify that both partition views share one budget. `getMaxWeight()` reports each cache's effective limit, so these two 1 MB assertions still pass if `ConnectorMetadataCache` groups by the physical cache name instead of `partition_view`. That would give the MVCC and list caches separate buckets and let them retain about 2 MB under one 1 MB entry limit. Exercise admission through both caches with payloads that fit individually but exceed the limit together, then assert their combined retained weight stays within 1 MB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6092638630
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6092638773
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](fe) Port external metadata cache budget tests from branch-4.1 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68839:
URL: https://github.com/apache/doris/pull/68839#issuecomment-6093244933
<details>
<summary>TPC-DS: <b>Total hot run time: 152035 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3506202c47e3aa4ad188b12227c64afa8f83f029, data reload: false
query5 4306 594 438 438
query6 424 202 185 185
query7 4811 453 224 224
query8 317 189 175 175
query9 8765 4029 4023 4023
query10 465 307 264 264
query11 6028 3430 3206 3206
query12 155 86 87 86
query13 1249 446 310 310
query14 6552 4863 4553 4553
query14_1 4310 4289 4219 4219
query15 209 213 189 189
query16 942 448 416 416
query17 881 666 544 544
query18 2418 439 333 333
query19 202 169 147 147
query20 82 80 113 80
query21 210 133 107 107
query22 13100 13036 12767 12767
query23 13189 12530 12190 12190
query23_1 12294 12286 12223 12223
query24 7083 829 447 447
query24_1 455 465 451 451
query25 500 392 336 336
query26 1249 254 133 133
query27 2778 434 265 265
query28 4620 1923 1924 1923
query29 1532 570 427 427
query30 295 228 175 175
query31 880 756 641 641
query32 142 93 87 87
query33 501 290 236 236
query34 881 840 493 493
query35 738 750 648 648
query36 813 825 720 720
query37 127 102 88 88
query38 1795 1779 1698 1698
query39 730 734 694 694
query39_1 661 662 662 662
query40 219 114 93 93
query41 64 63 63 63
query42 83 83 88 83
query43 364 373 328 328
query44 1283 682 685 682
query45 176 180 163 163
query46 840 947 555 555
query47 2874 3030 2959 2959
query48 285 302 210 210
query49 561 389 326 326
query50 667 284 210 210
query51 10152 10604 10075 10075
query52 78 87 68 68
query53 182 214 149 149
query54 244 207 193 193
query55 75 71 61 61
query56 235 227 215 215
query57 1610 1597 1568 1568
query58 300 258 259 258
query59 2378 2382 2137 2137
query60 286 236 221 221
query61 152 142 150 142
query62 411 340 282 282
query63 197 160 166 160
query64 2733 945 760 760
query65 3433 3359 3354 3354
query66 1779 415 298 298
query67 20246 20277 20214 20214
query68 3084 988 602 602
query69 377 291 236 236
query70 942 846 812 812
query71 307 232 223 223
query72 2960 2761 2539 2539
query73 528 535 294 294
query74 4623 4507 4321 4321
query75 2326 2318 1986 1986
query76 2295 1001 585 585
query77 351 394 299 299
query78 9103 8934 8443 8443
query79 1000 830 501 501
query80 517 430 338 338
query81 527 322 282 282
query82 575 136 108 108
query83 295 197 177 177
query84 311 128 106 106
query85 809 425 370 370
query86 319 238 220 220
query87 2027 1956 1847 1847
query88 3551 2658 2638 2638
query89 338 296 253 253
query90 1890 178 181 178
query91 159 150 125 125
query92 95 85 86 85
query93 972 976 560 560
query94 458 318 262 262
query95 586 416 318 318
query96 660 511 227 227
query97 2436 2477 2325 2325
query98 155 149 147 147
query99 781 774 634 634
Total cold run time: 233679 ms
Total hot run time: 152035 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org