You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "FMX (via GitHub)" <gi...@apache.org> on 2026/08/24 11:45:37 UTC

[PR] [CELEBORN-2442] Optimize meta memory for very large partitions [celeborn]

FMX opened a new pull request, #3825:
URL: https://github.com/apache/celeborn/pull/3825

   ### What changes were proposed in this pull request?
   
     This PR reduces the memory footprint of `PartitionLocation` metadata while preserving the existing wire protocol and public getter behavior.
   
     The main changes are:
   
     - Introduce an immutable, weakly interned `WorkerEndpoint` to share worker host and port information across partition locations.
     - Move the cached `host:pushPort` and `host:fetchPort` strings into the shared endpoint.
     - Lazily initialize `StorageInfo` and `RoaringBitmap` using thread-safe initialization.
     - Keep `getMapIdBitMap()` non-null for compatibility, and add a non-materializing accessor for serialization, HTTP API, and read-side inspection.
     - Avoid allocating empty bitmaps when locations are created by `LifecycleManager` and `SlotsAllocator`.
     - Preserve the current `storageTypes.head` selection behavior in `LifecycleManager`.
     - Keep string keys in `WorkerPartitionLocationInfo` to avoid parsing and boxing overhead on the Worker hot path.
     - Add allocation-free helpers for converting partition location IDs into packed `long` values.
     - Avoid materializing optional metadata from logging, protobuf serialization, HTTP rendering, and range-read paths.
     - Re-intern `WorkerEndpoint` instances after same-version Java deserialization.
     - Fix peer bitmap serialization so that the peer protobuf contains the peer's bitmap instead of the primary location's bitmap.
     - Make range reads fall back to the peer bitmap and fail open when bitmap metadata is unavailable.
     - Add JOL-based manual memory benchmarks and focused unit tests.
   
     No protobuf schema is changed by this PR.
   
     ### Why are the changes needed?
   
     A shuffle may contain millions of `PartitionLocation` instances. Previously, every instance independently retained:
   
     - Worker host and port fields.
     - Cached host/port strings.
     - An empty `StorageInfo`.
     - An empty `RoaringBitmap`.
   
     Most of this metadata is identical across locations or unused for the majority of their lifetime. The eager allocations therefore contribute significant heap usage and GC pressure on the Master, Worker, and client.
   
     The JOL benchmark includes both the live location graph and the weak interner overhead:
   
     | Scenario | Before | After, including interner | Reduction |
     |---|---:|---:|---:|
     | 10,000 primary/replica pairs across 2,000 endpoints | 5,672,304 bytes | 1,187,928 bytes | 79.06% |
     | 1,000,000 primary/replica pairs across 2,000 endpoints | 567,200,304 bytes | 88,307,928 bytes | 84.43% |
   
     The additional serialization and range-read fixes are needed to ensure that lazy bitmap allocation does not introduce data loss, incorrect peer metadata, or compatibility regressions.
   
     ### Does this PR resolve a correctness bug?
   
     <!-- Check if yes. The `correctness` label will be added/removed automatically. -->
     - [x] Yes
   
     It fixes peer bitmap serialization and ensures that missing bitmap metadata does not cause an NPE or silently exclude a partition location during range reads.
   
     ### Does this PR introduce _any_ user-facing change?
   
     <!-- Check if yes. -->
     - [ ] Yes
   
     There are no configuration, wire-protocol, or intended behavioral changes for users.
   
     ### How was this patch tested?
   
     The following verification was completed:
   
     - Ran the `common`, `client`, and `master` module regression suites:
       - 366 JUnit tests, with 1 ignored.
       - 325 ScalaTest tests.
       - 691 tests in total, with no failures or errors.
     - Ran the final `PartitionLocationSuiteJ`:
       - 20 tests passed.
       - Covers lazy initialization, endpoint interning, endpoint setters, ID parsing, peer references, compatibility getters, and same-version Java serialization.
     - Ran Worker partition and storage regression suites:
       - 27 Java tests passed.
       - 9 Scala tests passed.
     - Ran the Spark 3.5 `LifecycleManagerReserveSlotsSuite` integration test:
       - 1 test passed.
       - All 11 reactor modules completed successfully.
     - Compiled the affected modules through SBT:
       - `celeborn-common/Test/compile`
       - `celeborn-service/Compile/compile`
     - Ran `spotless:check` and `git diff --check`.
     - Manually ran `PartitionLocationMemorySuiteJ` with JOL for both the 10,000-pair and 1,000,000-pair scenarios.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2442] Optimize meta memory for very large partitions [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #3825:
URL: https://github.com/apache/celeborn/pull/3825#discussion_r3853658439


##########
common/src/test/java/org/apache/celeborn/common/protocol/PartitionLocationMemorySuiteJ.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.protocol;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+
+import org.junit.Ignore;
+import org.junit.Test;
+
+@Ignore("Manual JOL benchmark; run main to print retained-size comparisons.")
+public class PartitionLocationMemorySuiteJ {
+
+  private static final int ENDPOINT_COUNT = 2000;
+
+  public static void main(String[] args) throws Exception {
+    int pairCount = args.length > 0 && "largePeer".equals(args[0]) ? 1_000_000 : 10_000;
+    new PartitionLocationMemorySuiteJ().printPeerPairFootprint(pairCount, ENDPOINT_COUNT);
+  }
+
+  @Test
+  public void printPartitionLocationFootprint() throws Exception {
+    printPeerPairFootprint(10_000, ENDPOINT_COUNT);
+  }
+
+  private void printPeerPairFootprint(int pairCount, int endpointCount) throws Exception {
+    compare(
+        loadGraphLayout(),
+        pairCount + " partition peer pairs across " + endpointCount + " endpoints",
+        newOldLocationPairArray(pairCount, endpointCount),
+        newLocationPairArray(pairCount, endpointCount));
+  }
+
+  private PartitionLocation[] newLocationPairArray(int size, int endpointCount) {
+    PartitionLocation[] locations = new PartitionLocation[size * 2];
+    for (int i = 0; i < size; i++) {
+      int endpointIndex = i % endpointCount;
+      PartitionLocation primary = newLocation(i, endpointIndex, PartitionLocation.Mode.PRIMARY);
+      PartitionLocation replica = newLocation(i, endpointIndex, PartitionLocation.Mode.REPLICA);
+      primary.setPeer(replica);
+      replica.setPeer(primary);
+      locations[i * 2] = primary;
+      locations[i * 2 + 1] = replica;
+    }
+    return locations;
+  }
+
+  private PartitionLocation newLocation(int id, int endpointIndex, PartitionLocation.Mode mode) {
+    return new PartitionLocation(
+        id,
+        0,
+        "localhost-" + endpointIndex,
+        1001 + endpointIndex,
+        1002 + endpointIndex,
+        1003 + endpointIndex,
+        1004 + endpointIndex,
+        mode);
+  }
+
+  private PartitionLocationOld[] newOldLocationPairArray(int size, int endpointCount) {

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2442] Optimize meta memory for very large partitions [celeborn]

Posted by "yew1eb (via GitHub)" <gi...@apache.org>.
yew1eb commented on code in PR #3825:
URL: https://github.com/apache/celeborn/pull/3825#discussion_r3843710129


##########
common/src/main/java/org/apache/celeborn/common/protocol/PartitionLocation.java:
##########
@@ -243,32 +229,93 @@ public String getUniqueId() {
     return id + "-" + epoch;
   }
 
+  public long getUniqueIdLong() {
+    return toUniqueIdLong(id, epoch);
+  }
+
+  public static long toUniqueIdLong(int id, int epoch) {
+    return ((long) id << 32) | (epoch & 0xffffffffL);
+  }
+
+  public static long toUniqueIdLong(String uniqueId) {
+    Long packedUniqueId = tryToUniqueIdLong(uniqueId);
+    if (packedUniqueId == null) {
+      throw new IllegalArgumentException("Invalid partition location unique id: " + uniqueId);
+    }
+    return packedUniqueId;
+  }
+
+  public static Long tryToUniqueIdLong(String uniqueId) {
+    if (uniqueId == null || uniqueId.length() < 3) {
+      return null;
+    }
+    int idStart = uniqueId.charAt(0) == '-' ? 1 : 0;
+    int separatorIndex = uniqueId.indexOf('-', idStart);
+    if (separatorIndex < 0) {
+      return null;
+    }
+    long id = tryParseInt(uniqueId, 0, separatorIndex);
+    long epoch = tryParseInt(uniqueId, separatorIndex + 1, uniqueId.length());
+    if (id == INVALID_PARSED_INT || epoch == INVALID_PARSED_INT) {
+      return null;
+    }
+    return toUniqueIdLong((int) id, (int) epoch);
+  }

Review Comment:
   `getUniqueIdLong` / `toUniqueIdLong` / `tryToUniqueIdLong` (and `tryParseInt` below) have no production call site in this PR — `WorkerPartitionLocationInfo` keeps `String` keys. Shall we drop them to keep the protocol class minimal, or are they intended for a follow-up?



##########
common/pom.xml:
##########
@@ -158,6 +158,12 @@
       <artifactId>mockito-core</artifactId>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>org.openjdk.jol</groupId>
+      <artifactId>jol-core</artifactId>
+      <version>0.17</version>
+      <scope>test</scope>
+    </dependency>
     <dependency>

Review Comment:
   Nit: please move the version to `<dependencyManagement>` in the root `pom.xml` (e.g. a `jol.version` property), per project convention. Also, `jol-core` is not declared in `project/CelebornBuild.scala`, so the benchmark can't run under SBT — either add it there or note that it's Maven-only.



##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -172,6 +172,27 @@ public int partitionsRead() {
 
   public abstract int partitionsRead();
 
+  static boolean shouldSkipLocation(
+      boolean rangeReadFilter, int startMapIndex, int endMapIndex, PartitionLocation location) {
+    if (!rangeReadFilter || endMapIndex == Integer.MAX_VALUE) {
+      return false;
+    }
+    RoaringBitmap bitmap = location.getMapIdBitMapIfPresent();
+    if (bitmap == null && location.hasPeer()) {
+      bitmap = location.getPeer().getMapIdBitMapIfPresent();
+    }
+    if (bitmap == null) {
+      // Missing filter metadata cannot prove that this location is irrelevant.
+      return false;
+    }
+    for (int i = startMapIndex; i < endMapIndex; i++) {
+      if (bitmap.contains(i)) {
+        return false;
+      }

Review Comment:
   Nit (can be a follow-up): this per-index scan can be a single range query, O(containers) instead of O(range):
   
   ```java
   return !bitmap.intersects(startMapIndex, (long) endMapIndex);
   ```
   
   `intersects(long, long)` is available in RoaringBitmap 1.0.6.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2442] Optimize meta memory for very large partitions [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3825:
URL: https://github.com/apache/celeborn/pull/3825#discussion_r3845087192


##########
common/src/test/java/org/apache/celeborn/common/protocol/PartitionLocationMemorySuiteJ.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.protocol;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+
+import org.junit.Ignore;
+import org.junit.Test;
+
+@Ignore("Manual JOL benchmark; run main to print retained-size comparisons.")
+public class PartitionLocationMemorySuiteJ {
+
+  private static final int ENDPOINT_COUNT = 2000;
+
+  public static void main(String[] args) throws Exception {
+    int pairCount = args.length > 0 && "largePeer".equals(args[0]) ? 1_000_000 : 10_000;
+    new PartitionLocationMemorySuiteJ().printPeerPairFootprint(pairCount, ENDPOINT_COUNT);
+  }
+
+  @Test
+  public void printPartitionLocationFootprint() throws Exception {
+    printPeerPairFootprint(10_000, ENDPOINT_COUNT);
+  }
+
+  private void printPeerPairFootprint(int pairCount, int endpointCount) throws Exception {
+    compare(
+        loadGraphLayout(),
+        pairCount + " partition peer pairs across " + endpointCount + " endpoints",
+        newOldLocationPairArray(pairCount, endpointCount),
+        newLocationPairArray(pairCount, endpointCount));
+  }
+
+  private PartitionLocation[] newLocationPairArray(int size, int endpointCount) {
+    PartitionLocation[] locations = new PartitionLocation[size * 2];
+    for (int i = 0; i < size; i++) {
+      int endpointIndex = i % endpointCount;
+      PartitionLocation primary = newLocation(i, endpointIndex, PartitionLocation.Mode.PRIMARY);
+      PartitionLocation replica = newLocation(i, endpointIndex, PartitionLocation.Mode.REPLICA);
+      primary.setPeer(replica);
+      replica.setPeer(primary);
+      locations[i * 2] = primary;
+      locations[i * 2 + 1] = replica;
+    }
+    return locations;
+  }
+
+  private PartitionLocation newLocation(int id, int endpointIndex, PartitionLocation.Mode mode) {
+    return new PartitionLocation(
+        id,
+        0,
+        "localhost-" + endpointIndex,
+        1001 + endpointIndex,
+        1002 + endpointIndex,
+        1003 + endpointIndex,
+        1004 + endpointIndex,
+        mode);
+  }
+
+  private PartitionLocationOld[] newOldLocationPairArray(int size, int endpointCount) {

Review Comment:
   The headline JOL numbers are not production-shaped here. This 8-argument constructor leaves `storageInfo` null, while the old baseline always allocates a `StorageInfo`; however, both `LifecycleManager` and `SlotsAllocator` pass a non-null `StorageInfo`, and protobuf deserialization reconstructs one as well. Each call also creates a fresh `"localhost-" + endpointIndex` string, so the old graph retains one host string per location even though the allocator paths reuse `WorkerInfo.host`, whereas the new interner collapses those artificial duplicates. Please build both graphs from shared endpoint fixtures and equivalent non-null storage metadata (or add a production-shaped scenario) before publishing the 79–84% reduction.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2442] Optimize meta memory for very large partitions [celeborn]

Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on PR #3825:
URL: https://github.com/apache/celeborn/pull/3825#issuecomment-5675702913

   Thanks @FMX for the contribution!
   Merged to main(v1.0.0).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2442] Optimize meta memory for very large partitions [celeborn]

Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 closed pull request #3825: [CELEBORN-2442] Optimize meta memory for very large partitions
URL: https://github.com/apache/celeborn/pull/3825


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2442] Optimize meta memory for very large partitions [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #3825:
URL: https://github.com/apache/celeborn/pull/3825#discussion_r3853636464


##########
common/src/main/java/org/apache/celeborn/common/protocol/PartitionLocation.java:
##########
@@ -243,32 +229,93 @@ public String getUniqueId() {
     return id + "-" + epoch;
   }
 
+  public long getUniqueIdLong() {
+    return toUniqueIdLong(id, epoch);
+  }
+
+  public static long toUniqueIdLong(int id, int epoch) {
+    return ((long) id << 32) | (epoch & 0xffffffffL);
+  }
+
+  public static long toUniqueIdLong(String uniqueId) {
+    Long packedUniqueId = tryToUniqueIdLong(uniqueId);
+    if (packedUniqueId == null) {
+      throw new IllegalArgumentException("Invalid partition location unique id: " + uniqueId);
+    }
+    return packedUniqueId;
+  }
+
+  public static Long tryToUniqueIdLong(String uniqueId) {
+    if (uniqueId == null || uniqueId.length() < 3) {
+      return null;
+    }
+    int idStart = uniqueId.charAt(0) == '-' ? 1 : 0;
+    int separatorIndex = uniqueId.indexOf('-', idStart);
+    if (separatorIndex < 0) {
+      return null;
+    }
+    long id = tryParseInt(uniqueId, 0, separatorIndex);
+    long epoch = tryParseInt(uniqueId, separatorIndex + 1, uniqueId.length());
+    if (id == INVALID_PARSED_INT || epoch == INVALID_PARSED_INT) {
+      return null;
+    }
+    return toUniqueIdLong((int) id, (int) epoch);
+  }

Review Comment:
   I'll remove unused methods.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2442] Optimize meta memory for very large partitions [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3825:
URL: https://github.com/apache/celeborn/pull/3825#issuecomment-5404340483

   Thanks for the review. I'll update this PR soon. @yew1eb @SteNicholas 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org