You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/10/09 13:48:31 UTC
[PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
morningman opened a new pull request, #68842:
URL: https://github.com/apache/doris/pull/68842
### What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary:
**In short.** This ports four fixes from branch-4.1 #66717 (memory governance for external metadata caches) that master's re-implementation (#67726) does not have. Under a metadata cache weight limit, master still lets the Iceberg SDK keep its own unbounded manifest cache, fetches the whole snapshot history of REST `snapshot-loading-mode=refs` tables on every table cache miss, and under-weighs cached Paimon tables by 2-5x because they keep growing after admission. Catalogs without a weight limit are not affected.
**Background.** #66717 bounds the estimated retained memory of external metadata caches with the FE-wide `external_meta_cache_max_weight`, the catalog property `meta.cache.max-weight`, and per-entry `meta.cache.<engine>.<entry>.max-weight`. A value is weighed once, when it is admitted; if it does not fit, it is returned to the query but not cached. master re-implemented this in #67726 on the connector cache framework, and #68839 ports the 4.x tests. Comparing the two implementations found the gaps below; branch-4.1 handles all of them.
**The problem, and what it cost.**
- **Iceberg SDK manifest cache outside the limit.** `IcebergCatalogFactory` derives `io.manifest.cache-enabled=true` whenever `meta.cache.iceberg.manifest.*` is enabled. That SDK cache keeps up to `io.manifest.cache.max-total-bytes` (100 MB by default) of manifest content per FileIO, outside every Doris limit, and Glue / S3 Tables catalogs have one FileIO per table. A user who sets a limit gets a second, unbounded manifest cache next to the bounded one.
- **Remote IO while weighing REST refs-mode tables.** A weight-limited `IcebergTableCache` serializes the loaded metadata (`TableMetadataParser.toJson`) and weighs it. Both call `TableMetadata.snapshots()`, which in refs mode runs the catalog's supplier and fetches every snapshot. So each table cache miss pays one extra full-history REST load, exactly the cost refs mode exists to avoid, and the cached value then holds the whole history.
- **Paimon tables grow after admission.** `paimon-table` is weighed from the schema, location and catalog environment. A cached `FileStoreTable` then grows: the first `latestSnapshot()` or scan builds its transient store (`RowType` copies of the schema, a copy of the options), and field lookups build the four lazy index maps of every `RowType`. Measured on Paimon 1.4.2, the grown graph is 2-5x the admitted weight, so the heap the cache really holds can exceed `MAX_WEIGHT` by that factor.
- **Paimon REST data tokens are not charged.** With data tokens, `RESTCatalog` gives each table its own `RESTTokenFileIO`. The FileIO it delegates to lives in Paimon's process-wide cache, but the vended token it fetches on first data access (about 2 KB) belongs to the table and arrives after admission.
**How this PR fixes it.**
- `IcebergConnector#hasMetaCacheWeightLimit` decides whether any limit applies: the FE-global limit, the catalog limit, or the `max-weight` of one of the entries the connector reads (`table`, `partition`, `manifest`, `partition_view`, now one constant shared with the DDL validation). The three catalog option builders take that flag, and `appendManifestCacheProperties` does not derive `io.manifest.cache-enabled` under a limit. An explicit user value still wins, as in branch-4.1.
- `IcebergCacheSizeEstimator.areSnapshotsLoaded` reads `TableMetadata.snapshotsLoaded` (Iceberg exposes no accessor for it). A weight-limited `TableOwner` checks it first and, if the list is still behind its supplier, declines retention with `incomplete_estimate:iceberg_snapshots_not_loaded` instead of serializing. The statement keeps the live table, whose supplier still serves historical snapshots on demand.
- `PaimonCacheSizeEstimator` reserves the post-admission growth of every concrete `FileStoreTable` from its schema shape: a base, each field, each field nested in a row, each child type of an array / map / multiset / vector, each option and each key. The constants are calibrated against a retained-size oracle that walks the grown graph (new test helper `EstimatorCalibrationAssertions`). A table whose FileIO is a `RESTTokenFileIO` additionally reserves 4 KB for its token.
**Results.**
| Case | Before | After |
|---|---|---|
| `meta.cache.iceberg.manifest.enable=true` plus any weight limit | SDK manifest cache derived on, up to 100 MB per FileIO outside the limit | not derived; `io.manifest.cache-enabled=true` still turns it on |
| Weight-limited REST catalog, `snapshot-loading-mode=refs`, table cache miss | one extra remote load of the full snapshot list, full history retained | no extra load; table not retained, statement reads the live table |
| Paimon table with 301 INT columns, after one scan | 35 KB admitted, 114 KB retained | 134 KB admitted, 114 KB retained |
| Paimon table with a 200-field ROW column, after field lookups | 25 KB admitted, 117 KB retained | 143 KB admitted, 117 KB retained |
| Paimon REST data-token table | token (about 2 KB) not charged | 4 KB reserved |
Across 11 Paimon fixtures (append and primary-key tables, wide schemas, many options, nested rows, arrays and maps of rows, multisets) the admitted weight is now 1.2-1.7x the fully grown graph.
Not in this PR: wide schemas that exceed the 10,000-visit budget of `ReflectiveObjectSizeEstimator.estimateComplete` (for example HMS tables with about 1,000 primitive columns, or Paimon tables with about 3,400) are still never admitted under a limit. The same budget is shared by several connectors and the FE schema cache, so it needs its own change.
**Classes, and how they call each other.**
- `IcebergConnector`: `WEIGHTED_CACHE_ENTRIES`, `hasMetaCacheWeightLimit()`, and passes the flag at the three catalog builds.
- `IcebergCatalogFactory`: `buildBaseCatalogProperties`, `buildCatalogProperties` and `buildS3TablesCatalogProperties` take `metaCacheWeightLimited` and stay pure.
- `IcebergCatalogProperties`: validates `max-weight` keys against the shared entry list.
- `IcebergCacheSizeEstimator` / `IcebergTableCache.TableOwner`: snapshot probe and the guard before serialization.
- `PaimonCacheSizeEstimator`: `estimateStoreGrowth` and `estimateTableOwnedFileIO`.
```
IcebergConnector.createCatalog / createS3TablesCatalog / createDlfCatalog
└─ hasMetaCacheWeightLimit(): metaCache.hasEnclosingWeightLimit() || cacheSpec(entry).isWeightBounded()
└─ IcebergCatalogFactory.build*CatalogProperties(..., metaCacheWeightLimited)
└─ appendManifestCacheProperties: derive io.manifest.cache-enabled only when not limited
IcebergTableCache.borrow ─ loader ─ new TableOwner(table, ..., weighted)
└─ areSnapshotsLoaded(table) ? toJson + estimateTable : incomplete("iceberg_snapshots_not_loaded")
PaimonMetaCacheCatalog "paimon-table" size estimator
└─ PaimonCacheSizeEstimator.estimateTable ─ estimateFileStoreTable (each concrete table)
├─ estimateStoreGrowth(schema)
└─ estimateTableOwnedFileIO(fileIO)
```
Nothing changes for catalogs without a weight limit: the flag is false, and the probe and the reservations only run when an entry is weighed. Independent of #68839 (tests only); both add tests to `IcebergConnectorCacheTest` and `IcebergTableCacheTest`, in different places.
Original author: @CalvinKirs
### Release note
Under an external metadata cache weight limit, Iceberg catalogs no longer turn on the Iceberg SDK manifest content cache implicitly (set `io.manifest.cache-enabled=true` to keep it), weight-limited Iceberg REST catalogs using `snapshot-loading-mode=refs` no longer retain tables in the table cache, and Paimon table cache weights include what a cached table grows after admission.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `run-fe-ut.sh --run` on `IcebergCatalogFactoryTest`, `IcebergCatalogOptionsSnapshotTest`, `IcebergConnectorCacheTest`, `IcebergTableCacheTest`, `PaimonCacheSizeEstimatorTest` (new) and `PaimonMetaCacheCatalogTest`; reverting each production change turns exactly its new tests red
- all unit tests of fe-connector-paimon (678) and fe-connector-iceberg (1485): 0 failures
- `mvn checkstyle:check` on both modules: 0 violations
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- See the release note. All three changes apply only to catalogs with a metadata cache weight limit.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6088666189
<details>
<summary>TPC-DS: <b>Total hot run time: 151815 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce25717d42c556b34b106c533270e2658fed8961, data reload: false
query5 4321 578 457 457
query6 418 206 191 191
query7 4813 440 221 221
query8 321 180 166 166
query9 8752 3989 3977 3977
query10 476 319 250 250
query11 5919 3403 3181 3181
query12 144 85 81 81
query13 1258 426 311 311
query14 6492 4757 4454 4454
query14_1 4257 4235 4235 4235
query15 205 207 182 182
query16 943 433 425 425
query17 854 655 539 539
query18 2412 443 309 309
query19 202 159 143 143
query20 78 78 85 78
query21 211 128 112 112
query22 12952 12995 12874 12874
query23 13253 12552 12025 12025
query23_1 12216 12300 12291 12291
query24 7072 828 479 479
query24_1 463 467 476 467
query25 535 428 356 356
query26 1304 258 139 139
query27 2757 432 269 269
query28 4605 1916 1934 1916
query29 1574 595 462 462
query30 304 218 184 184
query31 899 767 651 651
query32 152 90 92 90
query33 508 300 244 244
query34 917 865 502 502
query35 753 785 633 633
query36 824 819 719 719
query37 139 103 93 93
query38 1789 1739 1706 1706
query39 722 693 686 686
query39_1 664 659 674 659
query40 218 121 107 107
query41 73 72 73 72
query42 88 84 86 84
query43 360 384 324 324
query44 1288 696 699 696
query45 185 182 166 166
query46 827 957 567 567
query47 3038 3041 2905 2905
query48 304 318 223 223
query49 583 424 290 290
query50 662 276 198 198
query51 10644 10400 10419 10400
query52 80 82 67 67
query53 189 208 150 150
query54 247 217 171 171
query55 73 66 60 60
query56 240 217 208 208
query57 1638 1581 1541 1541
query58 279 266 245 245
query59 2360 2371 2148 2148
query60 290 229 216 216
query61 148 150 143 143
query62 393 334 278 278
query63 195 163 163 163
query64 2726 955 764 764
query65 3415 3364 3354 3354
query66 1771 402 335 335
query67 20177 20057 20044 20044
query68 3008 984 587 587
query69 373 290 245 245
query70 898 840 865 840
query71 278 227 211 211
query72 2723 2306 2291 2291
query73 509 520 298 298
query74 4600 4464 4311 4311
query75 2332 2290 1911 1911
query76 2304 1001 591 591
query77 354 396 311 311
query78 9076 8901 8503 8503
query79 982 862 530 530
query80 1225 446 328 328
query81 591 316 273 273
query82 615 156 113 113
query83 298 192 176 176
query84 311 133 97 97
query85 845 425 363 363
query86 397 236 269 236
query87 2010 1937 1795 1795
query88 3665 2711 2682 2682
query89 344 296 253 253
query90 1903 183 162 162
query91 158 148 118 118
query92 91 88 84 84
query93 1017 971 559 559
query94 636 298 293 293
query95 582 335 317 317
query96 663 519 235 235
query97 2446 2431 2294 2294
query98 155 147 143 143
query99 782 761 642 642
Total cold run time: 234848 ms
Total hot run time: 151815 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68842:
URL: https://github.com/apache/doris/pull/68842#discussion_r4233507701
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java:
##########
@@ -98,10 +133,62 @@ private static long estimateFileStoreTable(FileStoreTable table, Set<Object> vis
((DelegatedFileStoreTable) table).wrapped(), visited));
}
bytes = add(bytes, estimateCompleteOnce(table.schema(), visited));
+ bytes = add(bytes, estimateStoreGrowth(table.schema()));
+ bytes = add(bytes, estimateTableOwnedFileIO(table.fileIO(), visited));
bytes = add(bytes, estimatePath(table.location(), visited));
return add(bytes, estimateCatalogEnvironment(table.catalogEnvironment(), visited));
}
+ /** Reserves the store and index graph this table builds after admission; no IO, no store access. */
+ private static long estimateStoreGrowth(TableSchema schema) {
+ long bytes = add(STORE_GROWTH_BASE_BYTES,
+ multiply(schema.options().size(), STORE_GROWTH_OPTION_BYTES));
+ long keys = (long) schema.partitionKeys().size() + schema.primaryKeys().size() + schema.bucketKeys().size();
+ bytes = add(bytes, multiply(keys, STORE_GROWTH_KEY_BYTES));
+ for (String primaryKey : schema.primaryKeys()) {
+ bytes = add(bytes, JvmSizeUtils.stringSize(primaryKey));
+ }
+ return add(bytes, estimateFieldGrowth(schema.fields(), STORE_GROWTH_FIELD_BYTES));
+ }
Review Comment:
[P2] Reserve partial-update merge-factory growth. A wide primary-key table with `merge-engine=partial-update`, a sequence group covering many columns, and `fields.default-aggregate-function=last_non_null_value` builds a retained merge factory on its first scan. That factory adds two per-field maps and a capturing supplier per aggregated field; subsequent use can populate the retained RowType's lazy field indexes as well. For sufficiently many fields, this graph can exceed the fixed 320-byte-per-field reserve after cache admission. Account for this merge-engine-specific growth or decline weighted retention, and cover a wide partial-update table in the grown-graph test.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java:
##########
@@ -98,10 +133,62 @@ private static long estimateFileStoreTable(FileStoreTable table, Set<Object> vis
((DelegatedFileStoreTable) table).wrapped(), visited));
}
bytes = add(bytes, estimateCompleteOnce(table.schema(), visited));
+ bytes = add(bytes, estimateStoreGrowth(table.schema()));
+ bytes = add(bytes, estimateTableOwnedFileIO(table.fileIO(), visited));
bytes = add(bytes, estimatePath(table.location(), visited));
return add(bytes, estimateCatalogEnvironment(table.catalogEnvironment(), visited));
}
+ /** Reserves the store and index graph this table builds after admission; no IO, no store access. */
+ private static long estimateStoreGrowth(TableSchema schema) {
+ long bytes = add(STORE_GROWTH_BASE_BYTES,
+ multiply(schema.options().size(), STORE_GROWTH_OPTION_BYTES));
+ long keys = (long) schema.partitionKeys().size() + schema.primaryKeys().size() + schema.bucketKeys().size();
+ bytes = add(bytes, multiply(keys, STORE_GROWTH_KEY_BYTES));
+ for (String primaryKey : schema.primaryKeys()) {
+ bytes = add(bytes, JvmSizeUtils.stringSize(primaryKey));
+ }
+ return add(bytes, estimateFieldGrowth(schema.fields(), STORE_GROWTH_FIELD_BYTES));
+ }
+
+ private static long estimateFieldGrowth(List<DataField> fields, long perFieldBytes) {
+ long bytes = multiply(fields.size(), perFieldBytes);
+ for (DataField field : fields) {
+ bytes = add(bytes, estimateTypeGrowth(field.type()));
+ }
+ return bytes;
+ }
+
+ private static long estimateTypeGrowth(DataType type) {
+ if (type instanceof RowType) {
+ return estimateFieldGrowth(((RowType) type).getFields(),
+ STORE_GROWTH_FIELD_BYTES + STORE_GROWTH_NESTED_FIELD_BYTES);
+ }
+ if (type instanceof ArrayType) {
+ return estimateChildTypeGrowth(((ArrayType) type).getElementType());
+ }
+ if (type instanceof MultisetType) {
+ return estimateChildTypeGrowth(((MultisetType) type).getElementType());
+ }
+ if (type instanceof VectorType) {
+ return estimateChildTypeGrowth(((VectorType) type).getElementType());
+ }
+ if (type instanceof MapType) {
+ MapType mapType = (MapType) type;
+ return add(estimateChildTypeGrowth(mapType.getKeyType()),
+ estimateChildTypeGrowth(mapType.getValueType()));
+ }
+ return 0L;
+ }
+
+ private static long estimateChildTypeGrowth(DataType child) {
+ return add(STORE_GROWTH_TYPE_NODE_BYTES, estimateTypeGrowth(child));
+ }
+
+ private static long estimateTableOwnedFileIO(FileIO fileIO, Set<Object> visited) {
+ return fileIO instanceof RESTTokenFileIO && visited.add(fileIO) ? REST_TOKEN_FILE_IO_BYTES : 0L;
+ }
Review Comment:
[P1] Account for table-owned FileIO variants. This branch charges only `RESTTokenFileIO`, but Paimon's default REST path creates a fresh `ResolvingFileIO` for each table; it copies catalog options and retains backend IOs in a private map after cache admission. REST external tables use that resolver even with data tokens enabled, and HMS table loads likewise create a `HadoopFileIO` with its own lazy map. Because `PaimonMetaCacheCatalog` weighs tables only once, a weight-limited catalog can retain more heap than `meta.cache.max-weight` permits. Charge or bound these per-table graphs, or decline weighted retention, and cover the REST/HMS paths in tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68842:
URL: https://github.com/apache/doris/pull/68842#discussion_r4236518995
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java:
##########
@@ -98,10 +133,62 @@ private static long estimateFileStoreTable(FileStoreTable table, Set<Object> vis
((DelegatedFileStoreTable) table).wrapped(), visited));
}
bytes = add(bytes, estimateCompleteOnce(table.schema(), visited));
+ bytes = add(bytes, estimateStoreGrowth(table.schema()));
+ bytes = add(bytes, estimateTableOwnedFileIO(table.fileIO(), visited));
bytes = add(bytes, estimatePath(table.location(), visited));
return add(bytes, estimateCatalogEnvironment(table.catalogEnvironment(), visited));
}
+ /** Reserves the store and index graph this table builds after admission; no IO, no store access. */
+ private static long estimateStoreGrowth(TableSchema schema) {
+ long bytes = add(STORE_GROWTH_BASE_BYTES,
+ multiply(schema.options().size(), STORE_GROWTH_OPTION_BYTES));
+ long keys = (long) schema.partitionKeys().size() + schema.primaryKeys().size() + schema.bucketKeys().size();
+ bytes = add(bytes, multiply(keys, STORE_GROWTH_KEY_BYTES));
+ for (String primaryKey : schema.primaryKeys()) {
+ bytes = add(bytes, JvmSizeUtils.stringSize(primaryKey));
+ }
+ return add(bytes, estimateFieldGrowth(schema.fields(), STORE_GROWTH_FIELD_BYTES));
+ }
+
+ private static long estimateFieldGrowth(List<DataField> fields, long perFieldBytes) {
+ long bytes = multiply(fields.size(), perFieldBytes);
+ for (DataField field : fields) {
+ bytes = add(bytes, estimateTypeGrowth(field.type()));
+ }
+ return bytes;
+ }
+
+ private static long estimateTypeGrowth(DataType type) {
+ if (type instanceof RowType) {
+ return estimateFieldGrowth(((RowType) type).getFields(),
+ STORE_GROWTH_FIELD_BYTES + STORE_GROWTH_NESTED_FIELD_BYTES);
+ }
+ if (type instanceof ArrayType) {
+ return estimateChildTypeGrowth(((ArrayType) type).getElementType());
+ }
+ if (type instanceof MultisetType) {
+ return estimateChildTypeGrowth(((MultisetType) type).getElementType());
+ }
+ if (type instanceof VectorType) {
+ return estimateChildTypeGrowth(((VectorType) type).getElementType());
+ }
+ if (type instanceof MapType) {
+ MapType mapType = (MapType) type;
+ return add(estimateChildTypeGrowth(mapType.getKeyType()),
+ estimateChildTypeGrowth(mapType.getValueType()));
+ }
+ return 0L;
+ }
+
+ private static long estimateChildTypeGrowth(DataType child) {
+ return add(STORE_GROWTH_TYPE_NODE_BYTES, estimateTypeGrowth(child));
+ }
+
+ private static long estimateTableOwnedFileIO(FileIO fileIO, Set<Object> visited) {
+ return fileIO instanceof RESTTokenFileIO && visited.add(fileIO) ? REST_TOKEN_FILE_IO_BYTES : 0L;
+ }
Review Comment:
You're right, and my previous reply overstated it: ce25717d42c charged the allowance only on the `FileStoreTable` path. In Paimon 1.4.2, `CatalogUtils.loadTable` gives an external table of any kind the catalog's external FileIO, and `RESTCatalog` binds that to `fileIOFromOptions`, a new `ResolvingFileIO` per table even with data tokens enabled.
b6db733bc69 charges format, object, Lance and Iceberg tables the same 16 KB allowance. `tablesOutsideTheFileStoreCarryTheFileIoAllowanceToo` builds each of the four on its own `ResolvingFileIO` and checks that the weight exceeds the allowance and, once the FileIO has built its backend, still covers the retained graph. Without the new charge a format table weighs 1,600 bytes against a 16,408-byte allowance, and the test fails.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java:
##########
@@ -161,14 +163,17 @@ public static Map<String, String> buildBaseCatalogProperties(Map<String, String>
* {@code io.manifest.cache-enabled} directly — derive it to {@code "true"} from the FE meta-cache
* spec ({@code meta.cache.iceberg.manifest.*}) using the same {@code enable && ttl != 0 &&
* capacity != 0} rule. Default-disabled (legacy {@code DEFAULT_ICEBERG_MANIFEST_CACHE_ENABLE}).
+ * Nothing is derived under a metadata cache weight limit: the SDK cache keeps up to
+ * {@code io.manifest.cache.max-total-bytes} of manifest content per FileIO outside that limit.
*/
- private static void appendManifestCacheProperties(Map<String, String> props, Map<String, String> opts) {
+ private static void appendManifestCacheProperties(Map<String, String> props, Map<String, String> opts,
+ boolean metaCacheWeightLimited) {
boolean hasExplicitEnabled = StringUtils.isNotBlank(props.get(CatalogProperties.IO_MANIFEST_CACHE_ENABLED));
copyIfPresent(props, opts, CatalogProperties.IO_MANIFEST_CACHE_ENABLED);
copyIfPresent(props, opts, CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS);
copyIfPresent(props, opts, CatalogProperties.IO_MANIFEST_CACHE_MAX_TOTAL_BYTES);
copyIfPresent(props, opts, CatalogProperties.IO_MANIFEST_CACHE_MAX_CONTENT_LENGTH);
- if (!hasExplicitEnabled) {
+ if (!hasExplicitEnabled && !metaCacheWeightLimited) {
Review Comment:
Agreed for the server defaults, fixed in 4728d698666. Under a weight limit, when the catalog does not set `io.manifest.cache-enabled` itself, the client properties now carry `false`. `ConfigResponse.merge` starts from the server defaults and puts the client properties on top, so a REST `/config` default of `true` no longer turns the cache on. `metaCacheWeightLimitTurnsTheSdkManifestCacheOff` merges the built properties with such a response; leaving the key out makes it fail.
Server overrides and per-table config are left alone. The REST protocol defines overrides as values the server mandates over client configuration, and `RESTSessionCatalog.tableFileIO` lets a table's config win over the catalog's. Forcing `false` past them would mean replacing the FileIO the SDK builds, and a server that mandates the cache has chosen to spend that memory. An `io.manifest.cache-enabled` set on the catalog still wins, as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68842:
URL: https://github.com/apache/doris/pull/68842#discussion_r4234105307
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java:
##########
@@ -161,14 +163,17 @@ public static Map<String, String> buildBaseCatalogProperties(Map<String, String>
* {@code io.manifest.cache-enabled} directly — derive it to {@code "true"} from the FE meta-cache
* spec ({@code meta.cache.iceberg.manifest.*}) using the same {@code enable && ttl != 0 &&
* capacity != 0} rule. Default-disabled (legacy {@code DEFAULT_ICEBERG_MANIFEST_CACHE_ENABLE}).
+ * Nothing is derived under a metadata cache weight limit: the SDK cache keeps up to
+ * {@code io.manifest.cache.max-total-bytes} of manifest content per FileIO outside that limit.
*/
- private static void appendManifestCacheProperties(Map<String, String> props, Map<String, String> opts) {
+ private static void appendManifestCacheProperties(Map<String, String> props, Map<String, String> opts,
+ boolean metaCacheWeightLimited) {
boolean hasExplicitEnabled = StringUtils.isNotBlank(props.get(CatalogProperties.IO_MANIFEST_CACHE_ENABLED));
copyIfPresent(props, opts, CatalogProperties.IO_MANIFEST_CACHE_ENABLED);
copyIfPresent(props, opts, CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS);
copyIfPresent(props, opts, CatalogProperties.IO_MANIFEST_CACHE_MAX_TOTAL_BYTES);
copyIfPresent(props, opts, CatalogProperties.IO_MANIFEST_CACHE_MAX_CONTENT_LENGTH);
- if (!hasExplicitEnabled) {
+ if (!hasExplicitEnabled && !metaCacheWeightLimited) {
Review Comment:
[P2] Keep the SDK manifest cache off after REST config merging. With a Doris weight limit and no explicit `io.manifest.cache-enabled`, this branch omits the key, but an Iceberg REST `/config` default of `true` fills it in `ConfigResponse.merge`. `RESTSessionCatalog` then initializes its FileIO with that value, and `ManifestFiles` can keep up to 100 MB of manifest content per FileIO outside the Doris limit. Enforce `false` on the effective FileIO properties, or reject a server-supplied `true`, unless Doris explicitly opted in; setting only the client map would still lose to server overrides or per-table config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6086790798
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6086791113
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68842:
URL: https://github.com/apache/doris/pull/68842#discussion_r4236815553
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java:
##########
@@ -98,10 +133,62 @@ private static long estimateFileStoreTable(FileStoreTable table, Set<Object> vis
((DelegatedFileStoreTable) table).wrapped(), visited));
}
bytes = add(bytes, estimateCompleteOnce(table.schema(), visited));
+ bytes = add(bytes, estimateStoreGrowth(table.schema()));
+ bytes = add(bytes, estimateTableOwnedFileIO(table.fileIO(), visited));
bytes = add(bytes, estimatePath(table.location(), visited));
return add(bytes, estimateCatalogEnvironment(table.catalogEnvironment(), visited));
}
+ /** Reserves the store and index graph this table builds after admission; no IO, no store access. */
+ private static long estimateStoreGrowth(TableSchema schema) {
+ long bytes = add(STORE_GROWTH_BASE_BYTES,
+ multiply(schema.options().size(), STORE_GROWTH_OPTION_BYTES));
+ long keys = (long) schema.partitionKeys().size() + schema.primaryKeys().size() + schema.bucketKeys().size();
+ bytes = add(bytes, multiply(keys, STORE_GROWTH_KEY_BYTES));
+ for (String primaryKey : schema.primaryKeys()) {
+ bytes = add(bytes, JvmSizeUtils.stringSize(primaryKey));
+ }
+ return add(bytes, estimateFieldGrowth(schema.fields(), STORE_GROWTH_FIELD_BYTES));
+ }
+
+ private static long estimateFieldGrowth(List<DataField> fields, long perFieldBytes) {
+ long bytes = multiply(fields.size(), perFieldBytes);
+ for (DataField field : fields) {
+ bytes = add(bytes, estimateTypeGrowth(field.type()));
+ }
+ return bytes;
+ }
+
+ private static long estimateTypeGrowth(DataType type) {
+ if (type instanceof RowType) {
+ return estimateFieldGrowth(((RowType) type).getFields(),
+ STORE_GROWTH_FIELD_BYTES + STORE_GROWTH_NESTED_FIELD_BYTES);
+ }
+ if (type instanceof ArrayType) {
+ return estimateChildTypeGrowth(((ArrayType) type).getElementType());
+ }
+ if (type instanceof MultisetType) {
+ return estimateChildTypeGrowth(((MultisetType) type).getElementType());
+ }
+ if (type instanceof VectorType) {
+ return estimateChildTypeGrowth(((VectorType) type).getElementType());
+ }
+ if (type instanceof MapType) {
+ MapType mapType = (MapType) type;
+ return add(estimateChildTypeGrowth(mapType.getKeyType()),
+ estimateChildTypeGrowth(mapType.getValueType()));
+ }
+ return 0L;
+ }
+
+ private static long estimateChildTypeGrowth(DataType child) {
+ return add(STORE_GROWTH_TYPE_NODE_BYTES, estimateTypeGrowth(child));
+ }
+
+ private static long estimateTableOwnedFileIO(FileIO fileIO, Set<Object> visited) {
+ return fileIO instanceof RESTTokenFileIO && visited.add(fileIO) ? REST_TOKEN_FILE_IO_BYTES : 0L;
+ }
Review Comment:
Thanks, the mechanism is right: `ResolvingFileIO.configure` copies `context.options()` into a fresh `Options` for every table, and `RESTCatalog.options()` is that same map, so each table carries about 40 bytes per catalog option (a map node and its bucket; the strings are shared with the catalog).
We are keeping the fixed allowance, as branch-4.x does. 16 KiB covers about 400 copied entries, and a Paimon catalog's options (the Doris `paimon.*` properties plus a REST server's `/config` defaults and overrides) are normally a few dozen. Reaching 1,024 takes a configuration that the operator or the catalog's own REST server would have to choose, and the overrun stays proportional to that count. Scaling the allowance with the option count would change branch-4.x's estimate as well, so it belongs in a separate PR for both branches if a deployment ever comes close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6082325575
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #68842:
URL: https://github.com/apache/doris/pull/68842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6090349244
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68842_ce25717d42c556b34b106c533270e2658fed8961_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68842_ce25717d42c556b34b106c533270e2658fed8961_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6083583971
<details>
<summary>ClickBench: <b>Total hot run time: 24.39 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b940b3512168ddee36fe0eb7816b558ecf1f0fb, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.12 0.11
query4 1.61 0.12 0.11
query5 0.27 0.24 0.24
query6 1.19 0.68 0.69
query7 0.03 0.01 0.00
query8 0.05 0.03 0.04
query9 0.43 0.35 0.35
query10 0.61 0.57 0.57
query11 0.20 0.15 0.15
query12 0.19 0.15 0.15
query13 0.49 0.48 0.50
query14 0.98 0.97 0.97
query15 0.64 0.61 0.60
query16 0.34 0.31 0.34
query17 1.11 1.13 1.13
query18 0.22 0.21 0.21
query19 2.09 1.93 1.97
query20 0.02 0.02 0.01
query21 15.50 0.27 0.14
query22 4.47 0.05 0.06
query23 16.24 0.31 0.12
query24 2.92 0.69 0.35
query25 0.12 0.05 0.07
query26 1.29 0.26 0.15
query27 0.04 0.04 0.03
query28 2.05 0.61 0.40
query29 12.51 4.10 3.26
query30 0.28 0.13 0.13
query31 2.77 0.58 0.35
query32 3.23 0.61 0.51
query33 3.14 3.22 3.22
query34 15.55 4.00 3.40
query35 3.35 3.35 3.36
query36 0.54 0.44 0.45
query37 0.10 0.06 0.07
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.17 0.15 0.14
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 95.37 s
Total hot run time: 24.39 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6088739913
<details>
<summary>ClickBench: <b>Total hot run time: 24.45 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce25717d42c556b34b106c533270e2658fed8961, data reload: false
query1 0.01 0.00 0.00
query2 0.09 0.05 0.05
query3 0.27 0.12 0.11
query4 1.62 0.12 0.12
query5 0.27 0.25 0.25
query6 1.18 0.69 0.67
query7 0.04 0.01 0.00
query8 0.06 0.03 0.04
query9 0.44 0.35 0.35
query10 0.59 0.60 0.58
query11 0.21 0.13 0.13
query12 0.19 0.15 0.14
query13 0.48 0.50 0.49
query14 0.98 0.96 0.97
query15 0.64 0.61 0.62
query16 0.31 0.36 0.35
query17 1.12 1.19 1.13
query18 0.22 0.20 0.20
query19 2.06 2.03 1.97
query20 0.02 0.01 0.01
query21 15.49 0.23 0.14
query22 4.51 0.06 0.05
query23 16.24 0.32 0.12
query24 3.10 0.67 0.42
query25 0.12 0.06 0.06
query26 1.50 0.25 0.15
query27 0.05 0.04 0.04
query28 2.20 0.62 0.42
query29 12.54 4.12 3.23
query30 0.28 0.14 0.13
query31 2.78 0.57 0.34
query32 3.22 0.61 0.52
query33 3.23 3.20 3.21
query34 15.45 4.01 3.38
query35 3.33 3.31 3.36
query36 0.55 0.45 0.45
query37 0.09 0.06 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.16 0.14 0.13
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 95.9 s
Total hot run time: 24.45 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6088056218
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6082233910
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6094105153
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6094105574
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68842:
URL: https://github.com/apache/doris/pull/68842#discussion_r4231053556
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java:
##########
@@ -46,8 +57,30 @@
* but their graphs are catalog-scoped executable services rather than entry-owned metadata. Walking
* those graphs both double-counts shared state and reaches strongly encapsulated JDK objects. The
* estimator therefore expands only immutable metadata owned by the entry.
+ *
+ * <p>A cached value is weighed once, at admission, but a {@link FileStoreTable} keeps growing afterwards: the
+ * first {@code latestSnapshot()} or scan builds its transient store (a {@code FileStore} holding its own
+ * {@code RowType} copies of the schema and a copy of the options) and field lookups build the four lazy index
+ * maps of every {@code RowType}. That growth is reserved from the schema's shape. On Paimon 1.4.2 it measures
+ * about 1-2 KB per table, 200-270 bytes per field, another 200 bytes per field nested in a row and 40 bytes per
+ * option; the constants below keep a margin above that.
*/
final class PaimonCacheSizeEstimator {
+ private static final long STORE_GROWTH_BASE_BYTES = 2048L;
+ private static final long STORE_GROWTH_FIELD_BYTES = 320L;
+ private static final long STORE_GROWTH_NESTED_FIELD_BYTES = 256L;
+ // A child type of an array, map, multiset or vector: the store's copy of it.
+ private static final long STORE_GROWTH_TYPE_NODE_BYTES = 128L;
+ private static final long STORE_GROWTH_OPTION_BYTES = 64L;
+ // A partition, primary or bucket key: its slot in the derived key types; a primary key is also copied as a
+ // renamed key field.
+ private static final long STORE_GROWTH_KEY_BYTES = 192L;
+ // RESTCatalog with data tokens gives each table its own RESTTokenFileIO. The FileIO it delegates to lives in
+ // Paimon's process-wide cache, but the vended token it fetches on first data access (an STS key id, secret and
+ // security token, about 2 KB) belongs to the table and arrives after admission.
+ private static final long REST_TOKEN_FILE_IO_BYTES =
+ JvmSizeUtils.saturatedAdd(JvmSizeUtils.instanceSize(RESTTokenFileIO.class), 4096L);
Review Comment:
[P2] Account for REST tokens larger than the fixed reserve. With `paimon.data-token.enabled=true` and a metadata weight limit, the REST table is admitted before scan planning calls `validToken()` on its retained `RESTTokenFileIO`. Paimon 1.4.2 stores the server's unrestricted token map there, and this 4 KB allowance is never reweighed. A credential larger than the allowance leaves the table cached below its actual retained weight; across many tables, `meta.cache.max-weight` can be exceeded. Please reweigh token replacement or decline weighted retention for token-bearing tables, and cover a long first-fetch or refresh token in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6083327679
<details>
<summary>TPC-H: <b>Total hot run time: 29220 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b940b3512168ddee36fe0eb7816b558ecf1f0fb, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17585 5570 5541 5541
q2 2119 327 267 267
q3 10197 1415 849 849
q4 4680 469 348 348
q5 7420 802 549 549
q6 189 190 164 164
q7 761 795 619 619
q8 9253 1367 1093 1093
q9 5665 4447 4474 4447
q10 6828 1288 1015 1015
q11 432 258 236 236
q12 635 403 285 285
q13 18054 3090 2388 2388
q14 270 279 248 248
q15 q16 731 727 682 682
q17 1183 744 576 576
q18 7075 6232 6171 6171
q19 1113 948 600 600
q20 382 337 230 230
q21 5313 2604 2714 2604
q22 411 362 308 308
Total cold run time: 100296 ms
Total hot run time: 29220 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6632 6578 6800 6578
q2 770 634 586 586
q3 5618 5640 5342 5342
q4 2087 2164 1577 1577
q5 5656 5530 5847 5530
q6 255 189 149 149
q7 2236 2049 1804 1804
q8 3239 2711 2822 2711
q9 8236 8183 8095 8095
q10 3635 3581 3225 3225
q11 596 401 371 371
q12 660 698 508 508
q13 2734 3064 2379 2379
q14 276 288 268 268
q15 q16 661 693 606 606
q17 7866 7026 6857 6857
q18 13170 12272 13105 12272
q19 906 784 782 782
q20 2181 2194 1914 1914
q21 5850 4716 4792 4716
q22 502 480 421 421
Total cold run time: 73766 ms
Total hot run time: 66691 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68842:
URL: https://github.com/apache/doris/pull/68842#discussion_r4233280480
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java:
##########
@@ -46,8 +57,30 @@
* but their graphs are catalog-scoped executable services rather than entry-owned metadata. Walking
* those graphs both double-counts shared state and reaches strongly encapsulated JDK objects. The
* estimator therefore expands only immutable metadata owned by the entry.
+ *
+ * <p>A cached value is weighed once, at admission, but a {@link FileStoreTable} keeps growing afterwards: the
+ * first {@code latestSnapshot()} or scan builds its transient store (a {@code FileStore} holding its own
+ * {@code RowType} copies of the schema and a copy of the options) and field lookups build the four lazy index
+ * maps of every {@code RowType}. That growth is reserved from the schema's shape. On Paimon 1.4.2 it measures
+ * about 1-2 KB per table, 200-270 bytes per field, another 200 bytes per field nested in a row and 40 bytes per
+ * option; the constants below keep a margin above that.
*/
final class PaimonCacheSizeEstimator {
+ private static final long STORE_GROWTH_BASE_BYTES = 2048L;
+ private static final long STORE_GROWTH_FIELD_BYTES = 320L;
+ private static final long STORE_GROWTH_NESTED_FIELD_BYTES = 256L;
+ // A child type of an array, map, multiset or vector: the store's copy of it.
+ private static final long STORE_GROWTH_TYPE_NODE_BYTES = 128L;
+ private static final long STORE_GROWTH_OPTION_BYTES = 64L;
+ // A partition, primary or bucket key: its slot in the derived key types; a primary key is also copied as a
+ // renamed key field.
+ private static final long STORE_GROWTH_KEY_BYTES = 192L;
+ // RESTCatalog with data tokens gives each table its own RESTTokenFileIO. The FileIO it delegates to lives in
+ // Paimon's process-wide cache, but the vended token it fetches on first data access (an STS key id, secret and
+ // security token, about 2 KB) belongs to the table and arrives after admission.
+ private static final long REST_TOKEN_FILE_IO_BYTES =
+ JvmSizeUtils.saturatedAdd(JvmSizeUtils.instanceSize(RESTTokenFileIO.class), 4096L);
Review Comment:
Partly agree. Paimon 1.4.2 keeps one `RESTToken` per `RESTTokenFileIO`, and `refreshToken()` replaces it, so the retained size is bounded by the largest single token; refreshes don't accumulate. Reweighing on replacement would need a hook Paimon doesn't offer. Declining weighted retention for token-bearing tables would turn the table cache off for every REST data-token catalog that has a weight limit.
So the reserve stays fixed, but 4 KB was only about twice a typical STS token. de828c532f2 raises it to 16 KB, the same allowance branch-4.x charges for a table's FileIO. The test now covers a first fetch and a refresh to an 8,192-character security token; the old 4 KB reserve fails that case (4,128 reserved vs 9,232 retained).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6094454725
<details>
<summary>ClickBench: <b>Total hot run time: 24.42 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4728d698666b74f2d839f4cdc57ac669d539811f, data reload: false
query1 0.00 0.00 0.01
query2 0.10 0.05 0.04
query3 0.26 0.12 0.12
query4 1.61 0.13 0.13
query5 0.27 0.24 0.24
query6 1.17 0.69 0.67
query7 0.04 0.01 0.01
query8 0.06 0.04 0.04
query9 0.42 0.35 0.36
query10 0.62 0.55 0.57
query11 0.20 0.15 0.14
query12 0.18 0.14 0.15
query13 0.50 0.50 0.48
query14 0.98 0.96 0.97
query15 0.64 0.60 0.61
query16 0.34 0.31 0.36
query17 1.14 1.16 1.13
query18 0.22 0.22 0.20
query19 2.10 1.96 1.95
query20 0.02 0.01 0.01
query21 15.49 0.28 0.14
query22 4.40 0.06 0.05
query23 16.27 0.32 0.13
query24 2.96 1.10 0.44
query25 0.10 0.05 0.05
query26 1.73 0.25 0.16
query27 0.05 0.03 0.03
query28 2.23 0.60 0.40
query29 12.48 4.15 3.30
query30 0.29 0.13 0.14
query31 2.76 0.58 0.34
query32 3.22 0.61 0.51
query33 3.28 3.31 3.14
query34 15.66 4.05 3.40
query35 3.34 3.36 3.35
query36 0.55 0.48 0.43
query37 0.09 0.07 0.06
query38 0.05 0.04 0.04
query39 0.03 0.03 0.03
query40 0.16 0.15 0.14
query41 0.09 0.04 0.02
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 96.18 s
Total hot run time: 24.42 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6094418389
<details>
<summary>TPC-DS: <b>Total hot run time: 151252 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4728d698666b74f2d839f4cdc57ac669d539811f, data reload: false
query5 4318 597 451 451
query6 419 203 191 191
query7 4807 448 222 222
query8 327 179 163 163
query9 8757 3933 3910 3910
query10 456 318 248 248
query11 5982 3408 3180 3180
query12 143 85 82 82
query13 1261 424 299 299
query14 6549 4737 4491 4491
query14_1 4221 4156 4224 4156
query15 208 197 182 182
query16 947 432 422 422
query17 842 662 558 558
query18 2412 414 314 314
query19 195 158 137 137
query20 92 82 78 78
query21 215 134 111 111
query22 12986 12988 12783 12783
query23 13068 12361 12088 12088
query23_1 12167 12080 12303 12080
query24 7094 806 464 464
query24_1 480 451 452 451
query25 507 389 339 339
query26 1295 252 136 136
query27 2759 433 264 264
query28 4604 1888 1927 1888
query29 1539 561 434 434
query30 296 220 180 180
query31 866 755 652 652
query32 130 89 82 82
query33 499 290 229 229
query34 878 856 517 517
query35 750 753 626 626
query36 801 820 761 761
query37 126 107 83 83
query38 1801 1736 1705 1705
query39 706 720 660 660
query39_1 669 656 668 656
query40 211 130 92 92
query41 64 62 61 61
query42 92 81 83 81
query43 369 369 320 320
query44 1276 675 671 671
query45 173 178 163 163
query46 837 939 557 557
query47 2956 3005 2943 2943
query48 293 295 208 208
query49 562 434 283 283
query50 663 270 207 207
query51 10466 10438 10059 10059
query52 76 77 70 70
query53 187 205 144 144
query54 231 200 173 173
query55 69 62 64 62
query56 230 222 217 217
query57 1611 1606 1556 1556
query58 290 256 249 249
query59 2369 2362 2118 2118
query60 290 237 219 219
query61 169 147 144 144
query62 390 341 284 284
query63 188 164 157 157
query64 2563 948 745 745
query65 3458 3361 3379 3361
query66 1750 405 296 296
query67 20317 20490 20391 20391
query68 3032 991 604 604
query69 380 277 243 243
query70 922 832 840 832
query71 283 221 213 213
query72 2727 2549 2253 2253
query73 513 515 303 303
query74 4590 4476 4301 4301
query75 2341 2315 1925 1925
query76 2302 998 614 614
query77 359 400 305 305
query78 8965 9025 8568 8568
query79 891 858 506 506
query80 544 434 328 328
query81 508 327 272 272
query82 208 127 105 105
query83 187 183 175 175
query84 284 128 98 98
query85 759 404 360 360
query86 255 235 231 231
query87 1998 1980 1804 1804
query88 3524 2666 2642 2642
query89 334 295 263 263
query90 2090 180 187 180
query91 152 150 121 121
query92 96 82 89 82
query93 979 981 546 546
query94 433 307 274 274
query95 583 338 377 338
query96 653 519 231 231
query97 2418 2420 2325 2325
query98 152 152 152 152
query99 778 766 654 654
Total cold run time: 232308 ms
Total hot run time: 151252 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6087903011
<details>
<summary>TPC-DS: <b>Total hot run time: 152019 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de828c532f25b88e7d828d9daccbf921a6470401, data reload: false
query5 4313 582 457 457
query6 429 208 185 185
query7 4816 455 247 247
query8 315 187 167 167
query9 8807 4040 4026 4026
query10 464 319 244 244
query11 5999 3418 3185 3185
query12 148 88 82 82
query13 1245 446 326 326
query14 6529 4830 4454 4454
query14_1 4290 4226 4265 4226
query15 210 191 182 182
query16 950 439 425 425
query17 856 661 533 533
query18 2420 433 314 314
query19 229 166 135 135
query20 82 79 78 78
query21 215 133 113 113
query22 12945 12986 12954 12954
query23 13132 12511 12146 12146
query23_1 12244 12174 12285 12174
query24 7107 818 466 466
query24_1 474 475 474 474
query25 538 419 364 364
query26 1264 267 138 138
query27 2749 436 272 272
query28 4582 1921 1927 1921
query29 1556 596 462 462
query30 307 210 185 185
query31 898 774 647 647
query32 145 94 89 89
query33 517 307 241 241
query34 911 861 493 493
query35 769 771 652 652
query36 816 790 723 723
query37 134 105 88 88
query38 1805 1770 1691 1691
query39 706 703 690 690
query39_1 674 668 653 653
query40 220 126 109 109
query41 75 70 67 67
query42 85 89 80 80
query43 358 375 326 326
query44 1354 681 697 681
query45 181 175 167 167
query46 854 978 579 579
query47 2979 2966 2911 2911
query48 294 300 209 209
query49 564 391 298 298
query50 677 268 212 212
query51 10300 10566 10424 10424
query52 80 76 69 69
query53 187 206 148 148
query54 258 203 175 175
query55 74 76 63 63
query56 241 209 209 209
query57 1626 1590 1541 1541
query58 279 253 245 245
query59 2362 2380 2110 2110
query60 268 233 216 216
query61 138 143 140 140
query62 398 343 290 290
query63 196 162 159 159
query64 2741 930 759 759
query65 3404 3341 3393 3341
query66 1762 404 291 291
query67 19966 20235 20139 20139
query68 3017 1041 623 623
query69 390 339 248 248
query70 955 903 851 851
query71 288 230 207 207
query72 2697 2620 2202 2202
query73 534 544 306 306
query74 4621 4513 4330 4330
query75 2367 2317 1942 1942
query76 2304 1011 636 636
query77 356 395 301 301
query78 9084 8953 8436 8436
query79 980 886 514 514
query80 533 423 345 345
query81 530 330 273 273
query82 575 142 113 113
query83 300 195 174 174
query84 316 128 108 108
query85 792 421 353 353
query86 312 240 229 229
query87 2007 1941 1829 1829
query88 3664 2704 2681 2681
query89 336 289 254 254
query90 1877 176 181 176
query91 152 150 123 123
query92 94 88 84 84
query93 1010 1031 562 562
query94 454 308 264 264
query95 588 337 331 331
query96 686 535 235 235
query97 2452 2422 2322 2322
query98 157 147 142 142
query99 786 768 652 652
Total cold run time: 233406 ms
Total hot run time: 152019 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6083499672
<details>
<summary>TPC-DS: <b>Total hot run time: 151271 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b940b3512168ddee36fe0eb7816b558ecf1f0fb, data reload: false
query5 4321 576 452 452
query6 425 203 185 185
query7 4809 458 218 218
query8 317 184 163 163
query9 8726 3963 3933 3933
query10 479 304 252 252
query11 5902 3442 3174 3174
query12 157 90 83 83
query13 1237 436 310 310
query14 6539 4729 4450 4450
query14_1 4232 4201 4191 4191
query15 208 201 186 186
query16 957 428 407 407
query17 883 657 559 559
query18 2417 429 312 312
query19 198 160 132 132
query20 81 78 80 78
query21 214 135 113 113
query22 12977 12999 12819 12819
query23 13130 12530 12213 12213
query23_1 12037 12151 12198 12151
query24 7091 819 452 452
query24_1 466 455 469 455
query25 507 390 334 334
query26 1308 255 135 135
query27 2757 433 263 263
query28 4631 1937 1922 1922
query29 1524 577 445 445
query30 298 218 177 177
query31 888 758 649 649
query32 138 93 87 87
query33 499 300 236 236
query34 907 850 514 514
query35 745 755 626 626
query36 847 793 731 731
query37 129 101 87 87
query38 1803 1727 1668 1668
query39 741 728 676 676
query39_1 660 671 652 652
query40 221 116 93 93
query41 66 63 66 63
query42 87 87 80 80
query43 357 383 322 322
query44 1311 686 683 683
query45 181 176 159 159
query46 848 959 550 550
query47 2858 3002 2938 2938
query48 305 298 209 209
query49 559 411 284 284
query50 695 280 202 202
query51 10475 10282 10342 10282
query52 86 82 71 71
query53 189 207 149 149
query54 230 253 180 180
query55 70 69 67 67
query56 219 226 218 218
query57 1642 1581 1554 1554
query58 276 256 248 248
query59 2370 2377 2115 2115
query60 297 239 220 220
query61 146 146 148 146
query62 391 335 284 284
query63 188 161 164 161
query64 2735 913 769 769
query65 3432 3341 3367 3341
query66 1778 414 298 298
query67 20363 20117 19916 19916
query68 2916 982 595 595
query69 378 288 246 246
query70 920 839 867 839
query71 299 229 207 207
query72 2744 2543 2256 2256
query73 522 517 293 293
query74 4572 4436 4279 4279
query75 2391 2308 1947 1947
query76 2298 1024 593 593
query77 359 400 298 298
query78 9110 8911 8503 8503
query79 877 847 512 512
query80 495 432 335 335
query81 525 325 267 267
query82 203 136 113 113
query83 192 188 185 185
query84 292 122 99 99
query85 762 431 368 368
query86 247 237 232 232
query87 1974 1995 1803 1803
query88 3565 2708 2685 2685
query89 328 294 253 253
query90 2116 178 178 178
query91 161 151 125 125
query92 94 90 87 87
query93 960 1011 565 565
query94 450 324 271 271
query95 579 400 328 328
query96 675 526 230 230
query97 2447 2412 2343 2343
query98 153 147 146 146
query99 765 762 656 656
Total cold run time: 232655 ms
Total hot run time: 151271 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6087977348
<details>
<summary>ClickBench: <b>Total hot run time: 24.3 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit de828c532f25b88e7d828d9daccbf921a6470401, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.12 0.12
query4 1.61 0.12 0.12
query5 0.25 0.26 0.25
query6 1.16 0.68 0.68
query7 0.04 0.01 0.01
query8 0.05 0.05 0.04
query9 0.44 0.37 0.35
query10 0.59 0.57 0.60
query11 0.21 0.14 0.14
query12 0.19 0.15 0.14
query13 0.49 0.49 0.48
query14 0.99 0.95 0.99
query15 0.63 0.61 0.60
query16 0.36 0.35 0.34
query17 1.09 1.14 1.12
query18 0.22 0.22 0.21
query19 2.09 1.96 1.96
query20 0.02 0.01 0.01
query21 15.42 0.28 0.15
query22 4.38 0.06 0.05
query23 16.26 0.32 0.12
query24 3.03 0.59 0.36
query25 0.12 0.05 0.05
query26 1.05 0.25 0.15
query27 0.04 0.04 0.03
query28 1.64 0.64 0.41
query29 12.51 4.12 3.25
query30 0.28 0.14 0.13
query31 2.77 0.57 0.33
query32 3.22 0.62 0.52
query33 3.08 3.19 3.14
query34 15.61 3.95 3.36
query35 3.33 3.37 3.33
query36 0.56 0.47 0.44
query37 0.10 0.06 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.04
query40 0.18 0.14 0.13
query41 0.09 0.03 0.02
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 94.64 s
Total hot run time: 24.3 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6087753393
<details>
<summary>TPC-H: <b>Total hot run time: 29214 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de828c532f25b88e7d828d9daccbf921a6470401, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17596 5664 5582 5582
q2 2133 327 262 262
q3 10219 1382 834 834
q4 4689 478 350 350
q5 7450 814 538 538
q6 191 196 166 166
q7 792 816 641 641
q8 9298 1342 1131 1131
q9 5817 4473 4464 4464
q10 6831 1293 1044 1044
q11 451 267 242 242
q12 643 406 290 290
q13 18031 3062 2382 2382
q14 287 271 248 248
q15 q16 728 726 680 680
q17 1195 741 599 599
q18 7048 6233 6174 6174
q19 1117 971 602 602
q20 394 339 222 222
q21 4836 2457 2597 2457
q22 414 369 306 306
Total cold run time: 100160 ms
Total hot run time: 29214 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6628 6500 6421 6421
q2 735 601 530 530
q3 5552 5792 5504 5504
q4 2078 2186 1628 1628
q5 5830 5637 5619 5619
q6 243 200 162 162
q7 2310 1985 1813 1813
q8 3162 2751 2864 2751
q9 8112 7718 7689 7689
q10 3659 3583 3220 3220
q11 602 413 401 401
q12 671 702 509 509
q13 2760 3062 2379 2379
q14 295 282 258 258
q15 q16 671 684 602 602
q17 7920 7150 6900 6900
q18 13144 12285 13051 12285
q19 922 809 804 804
q20 2189 2170 1948 1948
q21 5997 4829 4948 4829
q22 533 474 414 414
Total cold run time: 74013 ms
Total hot run time: 66666 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6090613254
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68842_ce25717d42c556b34b106c533270e2658fed8961/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68842_ce25717d42c556b34b106c533270e2658fed8961/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6094342956
<details>
<summary>TPC-H: <b>Total hot run time: 28963 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4728d698666b74f2d839f4cdc57ac669d539811f, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17577 5608 5527 5527
q2 2080 311 259 259
q3 10273 1400 838 838
q4 4668 472 348 348
q5 7429 792 535 535
q6 190 188 156 156
q7 755 795 609 609
q8 9247 1343 1088 1088
q9 5778 4413 4414 4413
q10 6823 1287 1029 1029
q11 435 254 238 238
q12 634 403 283 283
q13 18044 3045 2390 2390
q14 278 264 242 242
q15 q16 723 728 663 663
q17 1165 746 568 568
q18 7016 6196 6171 6171
q19 1098 944 592 592
q20 377 331 223 223
q21 5512 2492 2481 2481
q22 386 319 310 310
Total cold run time: 100488 ms
Total hot run time: 28963 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6432 6503 6415 6415
q2 726 600 566 566
q3 5559 5705 5224 5224
q4 2032 2127 1568 1568
q5 5805 5642 5814 5642
q6 248 191 142 142
q7 2356 1997 1826 1826
q8 3095 2746 2670 2670
q9 8296 8099 7751 7751
q10 3654 3602 3229 3229
q11 602 400 386 386
q12 675 699 496 496
q13 2759 3066 2391 2391
q14 289 291 286 286
q15 q16 663 686 619 619
q17 7818 6974 6842 6842
q18 13121 12229 13056 12229
q19 925 771 768 768
q20 2192 2185 1950 1950
q21 5714 4666 4736 4666
q22 500 474 397 397
Total cold run time: 73461 ms
Total hot run time: 66063 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6082236569
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68842:
URL: https://github.com/apache/doris/pull/68842#discussion_r4233945449
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java:
##########
@@ -98,10 +133,62 @@ private static long estimateFileStoreTable(FileStoreTable table, Set<Object> vis
((DelegatedFileStoreTable) table).wrapped(), visited));
}
bytes = add(bytes, estimateCompleteOnce(table.schema(), visited));
+ bytes = add(bytes, estimateStoreGrowth(table.schema()));
+ bytes = add(bytes, estimateTableOwnedFileIO(table.fileIO(), visited));
bytes = add(bytes, estimatePath(table.location(), visited));
return add(bytes, estimateCatalogEnvironment(table.catalogEnvironment(), visited));
}
+ /** Reserves the store and index graph this table builds after admission; no IO, no store access. */
+ private static long estimateStoreGrowth(TableSchema schema) {
+ long bytes = add(STORE_GROWTH_BASE_BYTES,
+ multiply(schema.options().size(), STORE_GROWTH_OPTION_BYTES));
+ long keys = (long) schema.partitionKeys().size() + schema.primaryKeys().size() + schema.bucketKeys().size();
+ bytes = add(bytes, multiply(keys, STORE_GROWTH_KEY_BYTES));
+ for (String primaryKey : schema.primaryKeys()) {
+ bytes = add(bytes, JvmSizeUtils.stringSize(primaryKey));
+ }
+ return add(bytes, estimateFieldGrowth(schema.fields(), STORE_GROWTH_FIELD_BYTES));
+ }
+
+ private static long estimateFieldGrowth(List<DataField> fields, long perFieldBytes) {
+ long bytes = multiply(fields.size(), perFieldBytes);
+ for (DataField field : fields) {
+ bytes = add(bytes, estimateTypeGrowth(field.type()));
+ }
+ return bytes;
+ }
+
+ private static long estimateTypeGrowth(DataType type) {
+ if (type instanceof RowType) {
+ return estimateFieldGrowth(((RowType) type).getFields(),
+ STORE_GROWTH_FIELD_BYTES + STORE_GROWTH_NESTED_FIELD_BYTES);
+ }
+ if (type instanceof ArrayType) {
+ return estimateChildTypeGrowth(((ArrayType) type).getElementType());
+ }
+ if (type instanceof MultisetType) {
+ return estimateChildTypeGrowth(((MultisetType) type).getElementType());
+ }
+ if (type instanceof VectorType) {
+ return estimateChildTypeGrowth(((VectorType) type).getElementType());
+ }
+ if (type instanceof MapType) {
+ MapType mapType = (MapType) type;
+ return add(estimateChildTypeGrowth(mapType.getKeyType()),
+ estimateChildTypeGrowth(mapType.getValueType()));
+ }
+ return 0L;
+ }
+
+ private static long estimateChildTypeGrowth(DataType child) {
+ return add(STORE_GROWTH_TYPE_NODE_BYTES, estimateTypeGrowth(child));
+ }
+
+ private static long estimateTableOwnedFileIO(FileIO fileIO, Set<Object> visited) {
+ return fileIO instanceof RESTTokenFileIO && visited.add(fileIO) ? REST_TOKEN_FILE_IO_BYTES : 0L;
+ }
Review Comment:
Agreed, fixed in ce25717d42c. A REST catalog without data tokens creates a `ResolvingFileIO` per table (`RESTCatalog.fileIOFromOptions`), and `HiveCatalog.fileIO(Path)` resolves one per table location; both build their backend maps after admission. Paimon offers no IO-free way to tell an owned FileIO from one the catalog shares, so every concrete table now carries the fixed 16 KB FileIO allowance, as on branch-4.x. Charging a shared FileIO to each table only makes the estimate conservative.
`everyTableCarriesAnAllowanceForItsFileIo` builds the same table on `LocalFileIO`, `ResolvingFileIO`, `HadoopFileIO` and `RESTTokenFileIO` and checks that each weight carries the allowance, and that the allowance covers a `ResolvingFileIO` once it has built its backend. Charging only `RESTTokenFileIO` again makes it fail.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCacheSizeEstimator.java:
##########
@@ -98,10 +133,62 @@ private static long estimateFileStoreTable(FileStoreTable table, Set<Object> vis
((DelegatedFileStoreTable) table).wrapped(), visited));
}
bytes = add(bytes, estimateCompleteOnce(table.schema(), visited));
+ bytes = add(bytes, estimateStoreGrowth(table.schema()));
+ bytes = add(bytes, estimateTableOwnedFileIO(table.fileIO(), visited));
bytes = add(bytes, estimatePath(table.location(), visited));
return add(bytes, estimateCatalogEnvironment(table.catalogEnvironment(), visited));
}
+ /** Reserves the store and index graph this table builds after admission; no IO, no store access. */
+ private static long estimateStoreGrowth(TableSchema schema) {
+ long bytes = add(STORE_GROWTH_BASE_BYTES,
+ multiply(schema.options().size(), STORE_GROWTH_OPTION_BYTES));
+ long keys = (long) schema.partitionKeys().size() + schema.primaryKeys().size() + schema.bucketKeys().size();
+ bytes = add(bytes, multiply(keys, STORE_GROWTH_KEY_BYTES));
+ for (String primaryKey : schema.primaryKeys()) {
+ bytes = add(bytes, JvmSizeUtils.stringSize(primaryKey));
+ }
+ return add(bytes, estimateFieldGrowth(schema.fields(), STORE_GROWTH_FIELD_BYTES));
+ }
Review Comment:
Agreed, fixed in ce25717d42c. I measured it on Paimon 1.4.2 with a 300-field primary-key table, one sequence group over every field and `fields.default-aggregate-function=last_non_null_value`. After a scan it retains about 460 bytes per field, about 150 more than the same table under deduplicate or aggregation, while the store reserve allowed 320. Partial-update tables now reserve another 256 bytes per field.
The grown-graph test now leaves the FileIO allowance out of the comparison, since that allowance pays for the FileIO and not the store, and it includes this table and an aggregation one. Without the new reserve the partial-update table is admitted at 139,256 bytes against a grown graph of 142,608, and the test fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6088056596
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep Iceberg and Paimon metadata caches within their weight limits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68842:
URL: https://github.com/apache/doris/pull/68842#issuecomment-6088520063
<details>
<summary>TPC-H: <b>Total hot run time: 28761 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce25717d42c556b34b106c533270e2658fed8961, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17625 5622 5540 5540
q2 2081 315 252 252
q3 10278 1390 841 841
q4 4678 468 343 343
q5 7422 812 527 527
q6 193 190 152 152
q7 766 812 616 616
q8 9250 1289 1053 1053
q9 5736 4453 4440 4440
q10 6830 1294 1021 1021
q11 425 249 234 234
q12 627 404 282 282
q13 18057 3042 2388 2388
q14 277 275 248 248
q15 q16 741 726 690 690
q17 1177 751 582 582
q18 7141 6226 6171 6171
q19 1243 954 598 598
q20 373 337 227 227
q21 5533 2286 2392 2286
q22 364 315 270 270
Total cold run time: 100817 ms
Total hot run time: 28761 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6143 6150 6104 6104
q2 688 595 541 541
q3 5253 5396 4961 4961
q4 2007 2127 1546 1546
q5 5320 5160 6033 5160
q6 254 205 158 158
q7 2214 2065 1863 1863
q8 3120 2789 2802 2789
q9 8218 8170 8264 8170
q10 3794 3750 3390 3390
q11 657 440 401 401
q12 704 717 547 547
q13 2929 3205 2538 2538
q14 320 315 283 283
q15 q16 718 722 687 687
q17 8359 7611 7430 7430
q18 13138 12287 13058 12287
q19 908 774 778 774
q20 2199 2195 1936 1936
q21 5760 4689 4751 4689
q22 487 441 414 414
Total cold run time: 73190 ms
Total hot run time: 66668 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org