You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "eolivelli (via GitHub)" <gi...@apache.org> on 2026/02/22 15:34:21 UTC

[PR] [CELEBORN-CELEBORN-2268] Improve test coverage for MEMORY and S3 storage [celeborn]

eolivelli opened a new pull request, #3608:
URL: https://github.com/apache/celeborn/pull/3608

   ### What changes were proposed in this pull request?
   This commit adds only tests and some useful debug information about using MEMORY and S3 storage.
   
   ### Why are the changes needed?
   Because there is not enough code coverage on some configurations that may happen in production,  in particular about:
   - using MEMORY storage
   - using only S3 storage
   - using MEMORY with eviction to S3
   
   There is an interesting case to test: when you configure MEMORY to S3 eviction and the dataset is small.
   It is important to ensure that no file is created in S3
   
   ### Does this PR resolve a correctness bug?
   
   No
   
   ### Does this PR introduce _any_ user-facing change?
   
   No
   
   ### How was this patch tested?
   
   It adds new integration tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2268] Improve test coverage for MEMORY and S3 storage [celeborn]

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on PR #3608:
URL: https://github.com/apache/celeborn/pull/3608#issuecomment-3949922263

   @SteNicholas @RexXiong can you please take a look ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2268] Improve test coverage for MEMORY and S3 storage [celeborn]

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on code in PR #3608:
URL: https://github.com/apache/celeborn/pull/3608#discussion_r2838157560


##########
tests/spark-it/src/test/scala/org/apache/celeborn/tests/spark/s3/BasicEndToEndTieredStorageTest.scala:
##########
@@ -67,9 +68,9 @@ class BasicEndToEndTieredStorageTest extends AnyFunSuite
 
     val s3url = container.getS3URL
     val augmentedConfiguration = Map(
-      CelebornConf.ACTIVE_STORAGE_TYPES.key -> "MEMORY,S3",

Review Comment:
   I changed this existing test that I contribute some time ago to test only "S3", MEMORY was not really meant to be tested, it is better to make the test simpler in order to prevent confusion to readers



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2268] Improve test coverage for MEMORY and S3 storage [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3608:
URL: https://github.com/apache/celeborn/pull/3608#issuecomment-3958121972

   @eolivelli, could you please rebase the latest main branch to resolve the conflicts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2268] Improve test coverage for MEMORY and S3 storage [celeborn]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3608:
URL: https://github.com/apache/celeborn/pull/3608#issuecomment-3943456233

   ## [Codecov](https://app.codecov.io/gh/apache/celeborn/pull/3608?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `0%` with `1 line` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 66.86%. Comparing base ([`2dd1b7a`](https://app.codecov.io/gh/apache/celeborn/commit/2dd1b7aac4d488c0496da1fc77cab9c79bccd69b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`33a6de3`](https://app.codecov.io/gh/apache/celeborn/commit/33a6de350286c7487a2b8e076200760f811ddfbf?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 19 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/celeborn/pull/3608?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...g/apache/celeborn/common/protocol/StorageInfo.java](https://app.codecov.io/gh/apache/celeborn/pull/3608?src=pr&el=tree&filepath=common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fceleborn%2Fcommon%2Fprotocol%2FStorageInfo.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vcHJvdG9jb2wvU3RvcmFnZUluZm8uamF2YQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/celeborn/pull/3608?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main    #3608      +/-   ##
   ==========================================
   - Coverage   67.13%   66.86%   -0.26%     
   ==========================================
     Files         357      357              
     Lines       21860    21967     +107     
     Branches     1943     1949       +6     
   ==========================================
   + Hits        14674    14687      +13     
   - Misses       6166     6264      +98     
   + Partials     1020     1016       -4     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/celeborn/pull/3608?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2268] Improve test coverage for MEMORY and S3 storage [celeborn]

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on code in PR #3608:
URL: https://github.com/apache/celeborn/pull/3608#discussion_r2841863492


##########
master/src/main/java/org/apache/celeborn/service/deploy/master/SlotsAllocator.java:
##########
@@ -260,8 +260,12 @@ private static StorageInfo getStorageInfo(
         storageInfo = new StorageInfo("", StorageInfo.Type.S3, availableStorageTypes);
       } else if (StorageInfo.OSSAvailable(availableStorageTypes)) {
         storageInfo = new StorageInfo("", StorageInfo.Type.OSS, availableStorageTypes);
-      } else {
+      } else if (StorageInfo.HDFSAvailable(availableStorageTypes)) {
         storageInfo = new StorageInfo("", StorageInfo.Type.HDFS, availableStorageTypes);
+      } else if (StorageInfo.memoryAvailable(availableStorageTypes)) {

Review Comment:
   I added unit tests to cover this method.
   this is indeed important because with the new unit tests we are sealing the behavior of the master choosing the storage types depending on the specs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-CELEBORN-2268] Improve test coverage for MEMORY and S3 storage [celeborn]

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on code in PR #3608:
URL: https://github.com/apache/celeborn/pull/3608#discussion_r2838135502


##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -1019,9 +1019,12 @@ private[celeborn] class Master(
       Utils.getSlotsPerDisk(slots.asInstanceOf[WorkerResource])
         .asScala.map { case (worker, slots) => worker.toUniqueId -> slots }.asJava,
       requestSlots.requestId)
-
+    val primaryLocationsByType = slots.values.asScala
+      .flatMap(entry => entry._1.asScala) // ._1 extracts the primary location
+      .groupBy(l => l.getStorageInfo.getType)
+      .mapValues(locations => locations.size)
     var offerSlotsMsg = s"Successfully offered slots for $numReducers reducers of $shuffleKey" +
-      s" on ${slots.size()} workers"
+      s" on ${slots.size()} workers, primary types: $primaryLocationsByType"

Review Comment:
   when you have different spark jobs that select different storageTypes this log line is very useful, to quickly check that the client is doing what it is expected to do



##########
master/src/main/java/org/apache/celeborn/service/deploy/master/SlotsAllocator.java:
##########
@@ -260,8 +260,12 @@ private static StorageInfo getStorageInfo(
         storageInfo = new StorageInfo("", StorageInfo.Type.S3, availableStorageTypes);
       } else if (StorageInfo.OSSAvailable(availableStorageTypes)) {
         storageInfo = new StorageInfo("", StorageInfo.Type.OSS, availableStorageTypes);
-      } else {
+      } else if (StorageInfo.HDFSAvailable(availableStorageTypes)) {
         storageInfo = new StorageInfo("", StorageInfo.Type.HDFS, availableStorageTypes);
+      } else if (StorageInfo.memoryAvailable(availableStorageTypes)) {

Review Comment:
   this change is not strictly needed by in some error conditions, when using MEMORY storage, you see PartitionLocations assigned to HDFS, even if HDFS is not configured.
   
   I can move this to another PR, but I think it's relevant to this patch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2268] Improve test coverage for MEMORY and S3 storage [celeborn]

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on PR #3608:
URL: https://github.com/apache/celeborn/pull/3608#issuecomment-3959282217

   Rebased


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2268] Improve test coverage for MEMORY and S3 storage [celeborn]

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on code in PR #3608:
URL: https://github.com/apache/celeborn/pull/3608#discussion_r2838135900


##########
master/src/main/java/org/apache/celeborn/service/deploy/master/SlotsAllocator.java:
##########
@@ -260,8 +260,12 @@ private static StorageInfo getStorageInfo(
         storageInfo = new StorageInfo("", StorageInfo.Type.S3, availableStorageTypes);
       } else if (StorageInfo.OSSAvailable(availableStorageTypes)) {
         storageInfo = new StorageInfo("", StorageInfo.Type.OSS, availableStorageTypes);
-      } else {
+      } else if (StorageInfo.HDFSAvailable(availableStorageTypes)) {
         storageInfo = new StorageInfo("", StorageInfo.Type.HDFS, availableStorageTypes);
+      } else if (StorageInfo.memoryAvailable(availableStorageTypes)) {

Review Comment:
   this change is not strictly needed by this patch
   but in some error conditions (in case I force the test to fail), when using MEMORY storage, you see PartitionLocations assigned to HDFS, even if HDFS is not configured.
   
   I can move this to another PR, but I think it's relevant to this patch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org