You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "dataroaring (via GitHub)" <gi...@apache.org> on 2026/04/03 04:05:36 UTC
[PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
dataroaring opened a new pull request, #62086:
URL: https://github.com/apache/doris/pull/62086
## Summary
- Introduce `CompactTabletMetaStore` using Structure-of-Arrays layout to replace per-tablet `TabletMeta` objects in `TabletInvertedIndex`
- Eliminates 16-byte Java object header per tablet, reducing per-tablet overhead from ~96 bytes to ~69 bytes (~28% savings, ~270 MB at 10M tablets)
- Uses `Long2IntOpenHashMap` for tabletId→slot mapping with 7 parallel primitive arrays and a free list for slot reuse
## Test plan
- [ ] `CompactTabletMetaStoreTest` covers add/get/remove, duplicate adds, storageMedium mutation, free list reuse, array growth, `toMap()`, `clear()`, all storage medium values
- [ ] Existing `TabletInvertedIndexTest` and `ClusterLoadStatisticsTest` pass (backward-compatible API via on-demand `TabletMeta` construction)
- [ ] No `tabletMetaMap` references remain in modified files — fully migrated to `tabletMetaStore`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4181719648
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4258243569
Addressed the review feedback in commit b50ad34:
**Comment #1 (getTabletMeta() GC concern):** Fixed the main hot-path usage — `buildPartitionInfoBySkew()` now uses individual field accessors (`getDbId`, `getTableId`, `getPartitionId`, `getIndexId`, `getStorageMedium`) instead of constructing a `TabletMeta` object per tablet. This loop iterates over all tablets (potentially millions), so avoiding per-iteration object allocation significantly reduces GC pressure. The remaining `getTabletMeta()` call at line 229 (tablet report handling) passes the full object to downstream methods and runs once per-BE, so it's appropriate to keep.
**Comment #2 (redundant containsKey in addTablet):** Already addressed in a prior revision — `addTablet()` now relies on `tabletMetaStore.add()` returning `false` for duplicates.
**Comment #3 (unused newSchemaHashes):** Already addressed in a prior revision — the `newSchemaHashes` array was removed.
**Comment #4 (getReplicaNumByBeIdAndStorageMedium):** Already addressed in a prior revision — the method already uses `tabletMetaStore.getStorageMedium(tabletId)` directly.
**Latent bug fix:** Also fixed a latent bug where `Preconditions.checkNotNull(tabletMeta)` ran after `tabletMeta` was already dereferenced (would NPE before reaching the check). Replaced with a `NOT_EXIST_VALUE` check on the first accessor call.
— ThinkOps 🤖
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4259652301
# FE Regression Coverage Report
Increment line coverage ` 58.20% (71/122)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4253402329
# FE UT Coverage Report
Increment line coverage ` 96.40% (107/111)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_411734555275724c4d03f99f54c83b68e34d61f2/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_411734555275724c4d03f99f54c83b68e34d61f2/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4261152874
# FE UT Coverage Report
Increment line coverage ` 92.62% (113/122)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4259820039
# FE Regression Coverage Report
Increment line coverage ` 58.20% (71/122)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-6159456999
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4261317633
# FE Regression Coverage Report
Increment line coverage ` 58.20% (71/122)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #62086: [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore
URL: https://github.com/apache/doris/pull/62086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4259802498
# FE Regression Coverage Report
Increment line coverage ` 58.20% (71/122)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4253734475
# FE Regression Coverage Report
Increment line coverage ` 63.96% (71/111)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_411734555275724c4d03f99f54c83b68e34d61f2_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_411734555275724c4d03f99f54c83b68e34d61f2_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4258967192
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4261558664
# FE Regression Coverage Report
Increment line coverage ` 58.20% (71/122)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4261485343
# FE Regression Coverage Report
Increment line coverage ` 58.20% (71/122)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4259816811
# FE Regression Coverage Report
Increment line coverage ` 58.20% (71/122)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4258242797
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4260474790
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #62086:
URL: https://github.com/apache/doris/pull/62086#discussion_r3031350072
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/CompactTabletMetaStore.java:
##########
@@ -0,0 +1,229 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.thrift.TStorageMedium;
+
+import it.unimi.dsi.fastutil.longs.Long2IntOpenHashMap;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Memory-compact storage for TabletMeta using Structure-of-Arrays layout.
+ *
+ * Instead of storing one TabletMeta object per tablet (each with a 16-byte Java object header),
+ * this class stores each field in a parallel primitive array indexed by an internal slot.
+ * A Long2IntOpenHashMap maps tabletId to the slot index.
+ *
+ * Deleted slots are reused via a free list embedded in the dbIds array.
+ *
+ * Thread safety: callers must hold appropriate locks (provided by TabletInvertedIndex).
+ */
+public class CompactTabletMetaStore {
+
+ private static final int INITIAL_CAPACITY = 1024;
+ private static final int ABSENT = -1;
+
+ // tabletId -> slot index
+ private Long2IntOpenHashMap tabletIdToSlot;
+
+ // parallel arrays indexed by slot
+ private long[] dbIds;
+ private long[] tableIds;
+ private long[] partitionIds;
+ private long[] indexIds;
+ private int[] oldSchemaHashes;
+ private int[] newSchemaHashes;
+ private byte[] storageMediumOrdinals;
+
Review Comment:
`newSchemaHashes` is allocated and grown alongside the other arrays, but it is never read or exposed, and `TabletMeta` currently has no getter for `newSchemaHash`. This adds per-tablet memory overhead and undermines the compaction goal.
Either remove `newSchemaHashes` entirely for now, or fully wire it through (capture the value on add and expose a getter / include it when constructing `TabletMeta`).
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java:
##########
@@ -126,10 +126,10 @@ public List<TabletMeta> getTabletMetaList(List<Long> tabletIdList) {
public void addTablet(long tabletId, TabletMeta tabletMeta) {
long stamp = writeLock();
try {
- if (tabletMetaMap.containsKey(tabletId)) {
+ if (tabletMetaStore.containsKey(tabletId)) {
return;
}
Review Comment:
`addTablet()` checks `tabletMetaStore.containsKey(tabletId)` and then calls `tabletMetaStore.add(tabletId, tabletMeta)`, but `CompactTabletMetaStore.add()` also checks for duplicates. This does two hash lookups for every add.
Consider removing the outer `containsKey()` check and rely on `tabletMetaStore.add()` to no-op on duplicates, or change `CompactTabletMetaStore.add()` to return a boolean and avoid duplicate lookups.
```suggestion
```
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java:
##########
@@ -101,7 +100,7 @@ public void tabletReport(long backendId, Map<Long, TTablet> backendTablets,
public TabletMeta getTabletMeta(long tabletId) {
long stamp = readLock();
try {
- return tabletMetaMap.get(tabletId);
+ return tabletMetaStore.getTabletMeta(tabletId);
} finally {
readUnlock(stamp);
}
Review Comment:
`getTabletMeta()` now constructs a new `TabletMeta` object on every call via `tabletMetaStore.getTabletMeta()`. `TabletInvertedIndex.getTabletMeta()` is used broadly across FE (rebalancers, report handler, proc nodes, etc.), so this can introduce significant allocation/GC overhead and may offset some of the memory win.
Consider adding allocation-free accessors on `TabletInvertedIndex`/`CompactTabletMetaStore` (e.g., `getDbId/getTableId/getPartitionId/getIndexId/getOldSchemaHash/getStorageMedium`) and migrating hot call sites to those, or providing a cached/flyweight `TabletMeta` for read-only access.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/LocalTabletInvertedIndex.java:
##########
@@ -896,7 +896,7 @@ public Map<TStorageMedium, Long> getReplicaNumByBeIdAndStorageMedium(long backen
Map<Long, Replica> replicaMetaWithBackend = backingReplicaMetaTable.get(backendId);
if (replicaMetaWithBackend != null) {
for (long tabletId : replicaMetaWithBackend.keySet()) {
- if (tabletMetaMap.get(tabletId).getStorageMedium() == TStorageMedium.HDD) {
+ if (tabletMetaStore.getTabletMeta(tabletId).getStorageMedium() == TStorageMedium.HDD) {
Review Comment:
In `getReplicaNumByBeIdAndStorageMedium()`, `tabletMetaStore.getTabletMeta(tabletId)` is called inside the loop solely to read `storageMedium`, which allocates a new `TabletMeta` per tablet. This is a hot path (iterating all tablets on a backend) and can add significant GC pressure.
Use `tabletMetaStore.getStorageMedium(tabletId)` instead (as is already done in `getTabletSizeByBackendIdAndStorageMedium`) to avoid per-iteration object allocation.
```suggestion
if (tabletMetaStore.getStorageMedium(tabletId) == TStorageMedium.HDD) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4252785900
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](memory) Replace TabletMeta object map with SoA CompactTabletMetaStore [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62086:
URL: https://github.com/apache/doris/pull/62086#issuecomment-4259201173
# FE Regression Coverage Report
Increment line coverage ` 58.20% (71/122)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62086_b50ad343b89faaf931c45220df8fbf185d372435_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org