You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "waitinfuture (via GitHub)" <gi...@apache.org> on 2024/06/09 11:55:37 UTC
[PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
waitinfuture opened a new pull request, #2549:
URL: https://github.com/apache/celeborn/pull/2549
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
https://github.com/apache/celeborn/pull/2348 avoids fetching first chunk in the constructor
of `CelebornInputStreamImpl`, but in some cases, i.e. coalescing 3000 partitions into one in Spark,
it can be beneficial to do so for performance. This PR adds back prefetching with knobs default to false.
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
Yes, two configs are added.
### How was this patch tested?
Extended `MemorySkewJoinSuite` and `ReusedExchangeSuite`, and manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2549: [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream
URL: https://github.com/apache/celeborn/pull/2549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#discussion_r1680310158
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4864,6 +4866,23 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(false)
+ val CLIENT_CHUNK_PREFETCH_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.client.chunk.prefetch.enabled")
Review Comment:
This config is checked inside `CelebornInputStreamImpl`, although it's used for spark only for now, I think we'd better not to add `spark` in the config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#discussion_r1680310962
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -214,46 +219,71 @@ class CelebornShuffleReader[K, C](
locations,
streamHandlers,
fileGroups.mapAttempts,
- metricsCallback)
+ metricsCallback,
+ chunkPrefetchEnabled)
+ streams.put(partitionId, inputStream)
} catch {
case e: IOException =>
logError(s"Exception caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, e)
- null
case e: Throwable =>
logError(s"Non IOException caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, new CelebornIOException(e))
- null
}
- } else null
+ }
}
+ (startPartition until Math.min(
Review Comment:
That's because when chunk prefetch is enabled, inputstream will create chunk buffer in construction, the purpose of window size is to control number of inputstreams to avoid OOM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#issuecomment-2227964242
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#issuecomment-2232213309
@RexXiong comments addressed, please take a look again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#discussion_r1680403295
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -214,46 +219,71 @@ class CelebornShuffleReader[K, C](
locations,
streamHandlers,
fileGroups.mapAttempts,
- metricsCallback)
+ metricsCallback,
+ chunkPrefetchEnabled)
+ streams.put(partitionId, inputStream)
} catch {
case e: IOException =>
logError(s"Exception caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, e)
- null
case e: Throwable =>
logError(s"Non IOException caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, new CelebornIOException(e))
- null
}
- } else null
+ }
}
+ (startPartition until Math.min(
Review Comment:
Adding a chunk limit that a task should prefetch to control the memory usage can be better.
16 may not be suitable for all scenarios. And this config will be hacky to tweak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#discussion_r1639955635
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -214,46 +219,71 @@ class CelebornShuffleReader[K, C](
locations,
streamHandlers,
fileGroups.mapAttempts,
- metricsCallback)
+ metricsCallback,
+ chunkPrefetchEnabled)
+ streams.put(partitionId, inputStream)
} catch {
case e: IOException =>
logError(s"Exception caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, e)
- null
case e: Throwable =>
logError(s"Non IOException caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, new CelebornIOException(e))
- null
}
- } else null
+ }
}
+ (startPartition until Math.min(
Review Comment:
Why we need a creation window for this?
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -187,7 +189,8 @@ private static final class CelebornInputStreamImpl extends CelebornInputStream {
int shuffleId,
int partitionId,
ExceptionMaker exceptionMaker,
- MetricsCallback metricsCallback)
+ MetricsCallback metricsCallback,
+ boolean chunkPrefetchEnabled)
Review Comment:
We can directly use conf.clientChunkPrefetchEnabled to get the value, instead of passing it from elsewhere and changing ShuffleClient/Impl method signature.
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4864,6 +4866,23 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(false)
+ val CLIENT_CHUNK_PREFETCH_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.client.chunk.prefetch.enabled")
Review Comment:
Only for spark? celeborn.client -> celeborn.client.spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#discussion_r1680403295
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -214,46 +219,71 @@ class CelebornShuffleReader[K, C](
locations,
streamHandlers,
fileGroups.mapAttempts,
- metricsCallback)
+ metricsCallback,
+ chunkPrefetchEnabled)
+ streams.put(partitionId, inputStream)
} catch {
case e: IOException =>
logError(s"Exception caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, e)
- null
case e: Throwable =>
logError(s"Non IOException caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, new CelebornIOException(e))
- null
}
- } else null
+ }
}
+ (startPartition until Math.min(
Review Comment:
Adding a chunk limit that a task should prefetch to control the memory usage can be better.
16 may not be suitable for all scenarios. And this config will be hacky to tweak.
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4864,6 +4866,23 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(false)
+ val CLIENT_CHUNK_PREFETCH_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.client.chunk.prefetch.enabled")
+ .categories("client")
+ .doc("Whether to enable chunk prefetch when creating CelebornInputStream.")
+ .version("0.6.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ val CLIENT_INPUTSTREAM_CREATION_WINDOW: ConfigEntry[Int] =
+ buildConf("celeborn.client.inputStream.creation.window")
+ .categories("client")
+ .doc(s"Window size that CelebornShuffleReader pre-creates CelebornInputStreams, for coalesced scenario" +
Review Comment:
Although this pr is suitable for coalesced scenarios, this is no limit for a task that reads multiple locations. I think this might cause unexpected high memory consumption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong closed pull request #2549: [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream
URL: https://github.com/apache/celeborn/pull/2549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#issuecomment-2210439835
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#discussion_r1636029741
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -214,46 +219,71 @@ class CelebornShuffleReader[K, C](
locations,
streamHandlers,
fileGroups.mapAttempts,
- metricsCallback)
+ metricsCallback,
+ chunkPrefetchEnabled)
+ streams.put(partitionId, inputStream)
} catch {
case e: IOException =>
logError(s"Exception caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, e)
- null
case e: Throwable =>
logError(s"Non IOException caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, new CelebornIOException(e))
- null
}
- } else null
+ }
}
+ (startPartition until Math.min(
+ startPartition + inputStreamCreationWindow,
+ endPartition)).foreach(partitionId => {
+ streamCreatorPool.submit(new Runnable {
+ override def run(): Unit = {
+ createInputStream(partitionId)
+ }
+ })
+ })
+
val recordIter = (startPartition until endPartition).iterator.map(partitionId => {
if (handle.numMappers > 0) {
val startFetchWait = System.nanoTime()
- val inputStream: CelebornInputStream = createInputStream(partitionId)
- if (exceptionRef.get() != null) {
- exceptionRef.get() match {
- case ce @ (_: CelebornIOException | _: PartitionUnRetryAbleException) =>
- if (throwsFetchFailure &&
- shuffleClient.reportShuffleFetchFailure(handle.shuffleId, shuffleId)) {
- throw new FetchFailedException(
- null,
- handle.shuffleId,
- -1,
- -1,
- partitionId,
- SparkUtils.FETCH_FAILURE_ERROR_MSG + handle.shuffleId + "/" + shuffleId,
- ce)
- } else
- throw ce
- case e => throw e
+ var inputStream: CelebornInputStream = streams.get(partitionId)
+ while (inputStream == null) {
+ if (exceptionRef.get() != null) {
+ exceptionRef.get() match {
+ case ce @ (_: CelebornIOException | _: PartitionUnRetryAbleException) =>
+ if (throwsFetchFailure &&
+ shuffleClient.reportShuffleFetchFailure(handle.shuffleId, shuffleId)) {
+ throw new FetchFailedException(
+ null,
+ handle.shuffleId,
+ -1,
+ -1,
+ partitionId,
+ SparkUtils.FETCH_FAILURE_ERROR_MSG + handle.shuffleId + "/" + shuffleId,
+ ce)
+ } else
+ throw ce
+ case e => throw e
+ }
}
+ logInfo("inputStream is null, sleeping...")
+ Thread.sleep(50)
Review Comment:
Why does this sleep 50 millseconds?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#discussion_r1636331106
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -214,46 +219,71 @@ class CelebornShuffleReader[K, C](
locations,
streamHandlers,
fileGroups.mapAttempts,
- metricsCallback)
+ metricsCallback,
+ chunkPrefetchEnabled)
+ streams.put(partitionId, inputStream)
} catch {
case e: IOException =>
logError(s"Exception caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, e)
- null
case e: Throwable =>
logError(s"Non IOException caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, new CelebornIOException(e))
- null
}
- } else null
+ }
}
+ (startPartition until Math.min(
+ startPartition + inputStreamCreationWindow,
+ endPartition)).foreach(partitionId => {
+ streamCreatorPool.submit(new Runnable {
+ override def run(): Unit = {
+ createInputStream(partitionId)
+ }
+ })
+ })
+
val recordIter = (startPartition until endPartition).iterator.map(partitionId => {
if (handle.numMappers > 0) {
val startFetchWait = System.nanoTime()
- val inputStream: CelebornInputStream = createInputStream(partitionId)
- if (exceptionRef.get() != null) {
- exceptionRef.get() match {
- case ce @ (_: CelebornIOException | _: PartitionUnRetryAbleException) =>
- if (throwsFetchFailure &&
- shuffleClient.reportShuffleFetchFailure(handle.shuffleId, shuffleId)) {
- throw new FetchFailedException(
- null,
- handle.shuffleId,
- -1,
- -1,
- partitionId,
- SparkUtils.FETCH_FAILURE_ERROR_MSG + handle.shuffleId + "/" + shuffleId,
- ce)
- } else
- throw ce
- case e => throw e
+ var inputStream: CelebornInputStream = streams.get(partitionId)
+ while (inputStream == null) {
+ if (exceptionRef.get() != null) {
+ exceptionRef.get() match {
+ case ce @ (_: CelebornIOException | _: PartitionUnRetryAbleException) =>
+ if (throwsFetchFailure &&
+ shuffleClient.reportShuffleFetchFailure(handle.shuffleId, shuffleId)) {
+ throw new FetchFailedException(
+ null,
+ handle.shuffleId,
+ -1,
+ -1,
+ partitionId,
+ SparkUtils.FETCH_FAILURE_ERROR_MSG + handle.shuffleId + "/" + shuffleId,
+ ce)
+ } else
+ throw ce
+ case e => throw e
+ }
}
+ logInfo("inputStream is null, sleeping...")
+ Thread.sleep(50)
Review Comment:
This is from original code, in fact such code is common in Celeborn's codebase.. No specific reason for 50ms, just a small period of time which I think not necessary for a config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#discussion_r1680313234
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -214,46 +219,71 @@ class CelebornShuffleReader[K, C](
locations,
streamHandlers,
fileGroups.mapAttempts,
- metricsCallback)
+ metricsCallback,
+ chunkPrefetchEnabled)
+ streams.put(partitionId, inputStream)
} catch {
case e: IOException =>
logError(s"Exception caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, e)
- null
case e: Throwable =>
logError(s"Non IOException caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, new CelebornIOException(e))
- null
}
- } else null
+ }
}
+ (startPartition until Math.min(
Review Comment:
It's related to https://github.com/apache/celeborn/pull/2348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#discussion_r1680401405
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4864,6 +4866,23 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(false)
+ val CLIENT_CHUNK_PREFETCH_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.client.chunk.prefetch.enabled")
+ .categories("client")
+ .doc("Whether to enable chunk prefetch when creating CelebornInputStream.")
+ .version("0.6.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ val CLIENT_INPUTSTREAM_CREATION_WINDOW: ConfigEntry[Int] =
+ buildConf("celeborn.client.inputStream.creation.window")
+ .categories("client")
+ .doc(s"Window size that CelebornShuffleReader pre-creates CelebornInputStreams, for coalesced scenario" +
Review Comment:
Although this pr is suitable for coalesced scenarios, this is no limit for a task that reads multiple locations. I think this might cause unexpected high memory consumption.
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -214,46 +219,71 @@ class CelebornShuffleReader[K, C](
locations,
streamHandlers,
fileGroups.mapAttempts,
- metricsCallback)
+ metricsCallback,
+ chunkPrefetchEnabled)
+ streams.put(partitionId, inputStream)
} catch {
case e: IOException =>
logError(s"Exception caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, e)
- null
case e: Throwable =>
logError(s"Non IOException caught when readPartition $partitionId!", e)
exceptionRef.compareAndSet(null, new CelebornIOException(e))
- null
}
- } else null
+ }
}
+ (startPartition until Math.min(
Review Comment:
Adding a limit that a task should prefetch to control the memory usage can be better.
16 may not be suitable for all scenarios. And this config will be hacky to tweak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1446] Enable chunk prefetch when initialize CelebornInputStream [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #2549:
URL: https://github.com/apache/celeborn/pull/2549#issuecomment-2236300314
Merge to main(v0.6.0) and branch-0.5(v0.5.1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org