You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "buska88 (via GitHub)" <gi...@apache.org> on 2025/02/07 06:51:54 UTC
[PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier … [celeborn]
buska88 opened a new pull request, #3090:
URL: https://github.com/apache/celeborn/pull/3090
…stage when fetch fail has been reported
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
for non barrier shuffle read stage, LifecycleManager#handleGetShuffleIdForApp always return appshuffleId whether fetch status is true or not.
### Why are the changes needed?
As described in [jira](https://issues.apache.org/jira/browse/CELEBORN-1855), If LifecycleManager only return appshuffleId whose fetch status is success, the task will fail directly to "there is no finished map stage associated with", and previous fetch fail event reported may not be fatal.So just give it a chance
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "buska88 (via GitHub)" <gi...@apache.org>.
buska88 commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2697014340
> > > @buska88 It would be better to check the validity of a shuffle Id after you get it instead of using a shuffle ID that marked as invalid.
> > > You can add the check here
> > > https://github.com/apache/celeborn/blob/6a836f9523b7eb03d3eff4bf1c23349bf8270064/client/src/main/java/org/apache/celeborn/client/ShuffleClientImpl.java#L612-L630
> >
> >
> > Sorry, I don't understand the purpose of checking shuffle id here. If the shuffle id is invalid, what should we do next?
>
> Maybe we can also throw FetchFailureException and wait task retry
OKay.I choose to throw FetchFailException in CelebornShuffleReader(shuffleclientImpl doesn't have spark dependency).And you can have a look at it when you have free time cc @FMX @RexXiong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2750484328
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2650114909
According to your Jira tickets "that shuffle fetch fails does not lead to stage fail because task speculation and another attempts succeed", I think the quoted scererio should not happen after the PR #3080 and #2921.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2661895212
@buska88 It would be better to check the validity of a shuffle Id after you get it instead of using a shuffle ID that marked as invalid.
You can add the check here https://github.com/apache/celeborn/blob/6a836f9523b7eb03d3eff4bf1c23349bf8270064/client/src/main/java/org/apache/celeborn/client/ShuffleClientImpl.java#L612-L630
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "buska88 (via GitHub)" <gi...@apache.org>.
buska88 commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2653606341
> > According to your Jira ticket, "that shuffle fetch fails does not lead to stage fail because task speculation and another attempts succeed", I think the quoted scenario should not happen if you have PR #3080 and #2921.
> > Do you have these two PRs for your Celeborn client?
>
> No.We mainly use branch-0.5.Would you please consider that merging these two pr into branch-0.5-rc?Those are important features and many users using branch-0.5 may need them.
As for this pr, it occurs to me that when a task of a stage throws fetchFail exception, in the small duration between fetchFail exception reported and the stage is aborted, following read tasks cans still get appId and finish shuffle reading.Then when stage retrying, these tasks may not re compute, which save resources.If following tasks fail due to failing to get appId, then they will be recomputed in the next stage-retry inevitable.
I think this pr has a tiny influence when we have those two prs you mentioned, so it seems an unnecessary pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2875534653
Thanks @buska88 Merge to main(v0.6.0) and also backport to branch-0.5(v0.5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "buska88 (via GitHub)" <gi...@apache.org>.
buska88 commented on code in PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#discussion_r2033328585
##########
client-spark/spark-3-4/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -77,7 +77,27 @@ class CelebornShuffleReader[K, C](
val serializerInstance = newSerializerInstance(dep)
- val shuffleId = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
+ var shuffleId = handle.shuffleId
+ try {
+ shuffleId = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
Review Comment:
Thanks for reviewing and i fix my code according to your views. cc @RexXiong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "buska88 (via GitHub)" <gi...@apache.org>.
buska88 commented on code in PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#discussion_r2043999787
##########
client-spark/spark-2/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -64,7 +64,23 @@ class CelebornShuffleReader[K, C](
val serializerInstance = dep.serializer.newInstance()
- val shuffleId = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
+ val (shuffleId, success) = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
+ if (!success) {
+ val ce = new UnsupportedOperationException(
+ s"unexpected! there is no finished map stage associated with appShuffleId ${handle.shuffleId}")
+ if (handle.throwsFetchFailure) {
+ throw new FetchFailedException(
Review Comment:
OKay i fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "buska88 (via GitHub)" <gi...@apache.org>.
buska88 commented on code in PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#discussion_r2083702950
##########
client-spark/spark-2/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -64,7 +64,24 @@ class CelebornShuffleReader[K, C](
val serializerInstance = dep.serializer.newInstance()
- val shuffleId = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
+ var shuffleId = handle.shuffleId
+ try {
+ shuffleId = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
Review Comment:
Thanks, fix that cc@cxzl25
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2689596166
> > @buska88 It would be better to check the validity of a shuffle Id after you get it instead of using a shuffle ID that marked as invalid.
> > You can add the check here
> > https://github.com/apache/celeborn/blob/6a836f9523b7eb03d3eff4bf1c23349bf8270064/client/src/main/java/org/apache/celeborn/client/ShuffleClientImpl.java#L612-L630
>
> Sorry, I don't understand the purpose of checking shuffle id here. If the shuffle id is invalid, what should we do next?
Maybe we can also throw FetchFailureException and wait task retry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "buska88 (via GitHub)" <gi...@apache.org>.
buska88 commented on code in PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#discussion_r2079448326
##########
client-spark/spark-2/src/main/java/org/apache/spark/shuffle/celeborn/SparkUtils.java:
##########
@@ -154,7 +154,7 @@ public static String getAppShuffleIdentifier(int appShuffleId, TaskContext conte
return appShuffleId + "-" + context.stageId() + "-" + context.stageAttemptNumber();
}
- public static int celebornShuffleId(
+ public static Tuple2<Integer, Boolean> celebornShuffleId(
Review Comment:
ok i will work on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#discussion_r2081456123
##########
client-spark/spark-2/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -64,7 +64,24 @@ class CelebornShuffleReader[K, C](
val serializerInstance = dep.serializer.newInstance()
- val shuffleId = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
+ var shuffleId = handle.shuffleId
+ try {
+ shuffleId = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
Review Comment:
```suggestion
val shuffleId = try {
SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
} catch {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "buska88 (via GitHub)" <gi...@apache.org>.
buska88 commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2653583624
> According to your Jira ticket, "that shuffle fetch fails does not lead to stage fail because task speculation and another attempts succeed", I think the quoted scenario should not happen if you have PR #3080 and #2921.
>
> Do you have these two PRs for your Celeborn client?
No.We mainly use branch-0.5.Would you please consider that merging these two pr into branch-0.5-rc?Those are important features and many users using branch-0.5 may need them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#discussion_r2017297875
##########
client-spark/spark-3-4/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -77,7 +77,27 @@ class CelebornShuffleReader[K, C](
val serializerInstance = newSerializerInstance(dep)
- val shuffleId = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
+ var shuffleId = handle.shuffleId
+ try {
+ shuffleId = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
Review Comment:
In this scenario, Celeborn would throw a CelebornException, so catching a CelebornIOException is incorrect. Moreover, I believe we should not catch the CelebornException itself, as we cannot identify the outcome when encountering it. I propose adding a `bool success` field to the `PbGetShuffleIdResponse` to indicate whether the shuffleId is available or not. This way, the client can throw a FetchFailureException if necessary, thereby improving the handling and expression of the situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong closed pull request #3090: [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported
URL: https://github.com/apache/celeborn/pull/3090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#discussion_r2076676843
##########
client-spark/spark-2/src/main/java/org/apache/spark/shuffle/celeborn/SparkUtils.java:
##########
@@ -154,7 +154,7 @@ public static String getAppShuffleIdentifier(int appShuffleId, TaskContext conte
return appShuffleId + "-" + context.stageId() + "-" + context.stageAttemptNumber();
}
- public static int celebornShuffleId(
+ public static Tuple2<Integer, Boolean> celebornShuffleId(
Review Comment:
nit: Should we consider throwing a CelebornRuntimeException when an unavailable celebornShuffleId is encountered? This can allows to catch the exception within the CelebornShuffleReader while maintaining the existing method signature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#discussion_r2043596515
##########
client-spark/spark-2/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:
##########
@@ -64,7 +64,23 @@ class CelebornShuffleReader[K, C](
val serializerInstance = dep.serializer.newInstance()
- val shuffleId = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
+ val (shuffleId, success) = SparkUtils.celebornShuffleId(shuffleClient, handle, context, false)
+ if (!success) {
+ val ce = new UnsupportedOperationException(
+ s"unexpected! there is no finished map stage associated with appShuffleId ${handle.shuffleId}")
+ if (handle.throwsFetchFailure) {
+ throw new FetchFailedException(
Review Comment:
```
client-spark/spark-2/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala:72: overloaded method constructor FetchFailedException with alternatives:
```
https://github.com/apache/celeborn/actions/runs/14331163433/job/40489322319?pr=3090#step:4:1295
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "buska88 (via GitHub)" <gi...@apache.org>.
buska88 commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2854107945
```
25/04/29 21:42:07 INFO dispatcher-CoarseGrainedScheduler TaskSetManager: Starting task 3928.1 in stage 23.0 (TID 29935, zw06-data-hdp-dn29550.mt, executor 5432, partition 3928, PROCESS_LOCAL, 8328 bytes)
25/04/29 21:42:07 WARN task-result-getter-3 TaskSetManager: Lost task 1434.0 in stage 23.0 (TID 26608, zw06-data-hdp-dn29550.mt, executor 5432): java.lang.OutOfMemoryError: GC overhead limit exceeded
25/04/29 21:42:07 INFO task-result-getter-3 TaskSetManager: Handle failed task, add task to pendingTasks, task 1434.0 in stage 23.0 (TID 26608, zw06-data-hdp-dn29550.mt, executor 5432)
25/04/29 21:42:07 INFO Reporter ApplicationMaster: AppMaster: targetNumExecutors=1400, pendingAllocate=0, runningExecutors=1400.
25/04/29 21:42:07 INFO dispatcher-BlockManagerMaster BlockManagerMasterEndpoint: Registering block manager zw06-data-hdp-dn27371.mt:26633 with 2004.6 MiB RAM, BlockManagerId(6004, zw06-data-hdp-dn27371.mt, 26633, None)
25/04/29 21:42:07 INFO dispatcher-CoarseGrainedScheduler TaskSetManager: Starting task 1434.1 in stage 23.0 (TID 29936, zw06-data-hdp-dn27371.mt, executor 6004, partition 1434, PROCESS_LOCAL, 8328 bytes)
25/04/29 21:42:07 INFO dispatcher-CoarseGrainedScheduler TaskSetManager: Starting task 3929.1 in stage 23.0 (TID 29937, zw06-data-hdp-dn27371.mt, executor 6004, partition 3929, PROCESS_LOCAL, 8328 bytes)
25/04/29 21:42:07 INFO dispatcher-CoarseGrainedScheduler TaskSetManager: Starting task 3927.1 in stage 23.0 (TID 29938, zw06-data-hdp-dn27371.mt, executor 6004, partition 3927, PROCESS_LOCAL, 8328 bytes)
25/04/29 21:42:08 INFO dispatcher-CoarseGrainedScheduler YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 5432.
25/04/29 21:42:08 INFO Reporter ApplicationMaster: AppMaster: targetNumExecutors=1400, pendingAllocate=0, runningExecutors=1400.
25/04/29 21:42:08 INFO dag-scheduler-event-loop DAGScheduler: Executor lost: 5432 (epoch 6)
25/04/29 21:42:08 INFO dispatcher-BlockManagerMaster BlockManagerMasterEndpoint: Trying to remove executor 5432 from BlockManagerMaster.
25/04/29 21:42:08 INFO dispatcher-BlockManagerMaster BlockManagerMasterEndpoint: Removing block manager BlockManagerId(5432, zw06-data-hdp-dn29550.mt, 17839, None)
25/04/29 21:42:08 INFO dag-scheduler-event-loop BlockManagerMaster: Removed 5432 successfully in removeExecutor
25/04/29 21:42:08 INFO dispatcher-CoarseGrainedScheduler YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.119.69.11:58434) with ID 5996
25/04/29 21:42:08 INFO spark-listener-group-executorManagement ExecutorMonitor: New executor 5996 has registered (new total is 1367)
25/04/29 21:42:08 INFO Reporter ApplicationMaster: AppMaster: targetNumExecutors=1400, pendingAllocate=0, runningExecutors=1400.
25/04/29 21:42:08 INFO dispatcher-BlockManagerMaster BlockManagerMasterEndpoint: Registering block manager zw06-data-hdp-dn29769.mt:22001 with 2004.6 MiB RAM, BlockManagerId(5996, zw06-data-hdp-dn29769.mt, 22001, None)
25/04/29 21:42:08 INFO dispatcher-CoarseGrainedScheduler TaskSetManager: Starting task 135.1 in stage 23.0 (TID 29939, zw06-data-hdp-dn29769.mt, executor 5996, partition 135, PROCESS_LOCAL, 8328 bytes)
25/04/29 21:42:08 INFO dispatcher-CoarseGrainedScheduler TaskSetManager: Starting task 136.1 in stage 23.0 (TID 29940, zw06-data-hdp-dn29769.mt, executor 5996, partition 136, PROCESS_LOCAL, 8328 bytes)
25/04/29 21:42:08 INFO dispatcher-CoarseGrainedScheduler TaskSetManager: Starting task 137.1 in stage 23.0 (TID 29941, zw06-data-hdp-dn29769.mt, executor 5996, partition 137, PROCESS_LOCAL, 8328 bytes)
25/04/29 21:42:08 INFO dispatcher-CoarseGrainedScheduler YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 4786.
25/04/29 21:42:08 INFO Reporter ApplicationMaster: AppMaster: targetNumExecutors=1400, pendingAllocate=0, runningExecutors=1400.
25/04/29 21:42:08 INFO dag-scheduler-event-loop DAGScheduler: Executor lost: 4786 (epoch 6)
25/04/29 21:42:08 INFO dispatcher-BlockManagerMaster BlockManagerMasterEndpoint: Trying to remove executor 4786 from BlockManagerMaster.
25/04/29 21:42:08 INFO dispatcher-BlockManagerMaster BlockManagerMasterEndpoint: Removing block manager BlockManagerId(4786, zw06-data-hdp-dn33950.mt, 35009, None)
25/04/29 21:42:08 INFO dag-scheduler-event-loop BlockManagerMaster: Removed 4786 successfully in removeExecutor
25/04/29 21:42:08 INFO Reporter ApplicationMaster: AppMaster: targetNumExecutors=1400, pendingAllocate=0, runningExecutors=1400.
25/04/29 21:42:08 INFO dispatcher-CoarseGrainedScheduler YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.119.150.13:51114) with ID 5991
25/04/29 21:42:08 INFO spark-listener-group-executorManagement ExecutorMonitor: New executor 5991 has registered (new total is 1368)
25/04/29 21:42:08 ERROR celeborn-dispatcher-110 SparkUtils: Can not get TaskSetManager for taskId: 29935
25/04/29 21:42:08 INFO celeborn-dispatcher-110 LifecycleManager: handle fetch failure for appShuffleId 5 shuffleId 5
```
Find a new case for this pr.Task 29935 launch on executor 5432.When executor 5432 lost, SparkUtils cannot get TaskSetManager by taskid, because taskSet.removeRunningTask(tid) has been called.So this pr seems useful for this case.The task can throw a fetchfail exception and re run the stage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "buska88 (via GitHub)" <gi...@apache.org>.
buska88 commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2671124521
> @buska88 It would be better to check the validity of a shuffle Id after you get it instead of using a shuffle ID that marked as invalid.
>
> You can add the check here
>
> https://github.com/apache/celeborn/blob/6a836f9523b7eb03d3eff4bf1c23349bf8270064/client/src/main/java/org/apache/celeborn/client/ShuffleClientImpl.java#L612-L630
Sorry, I don't understand the purpose of checking shuffle id here. If the shuffle id is invalid, what should we do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2661888490
> > > > According to your Jira ticket, "that shuffle fetch fails does not lead to stage fail because task speculation and another attempts succeed", I think the quoted scenario should not happen if you have PR #3080 and #2921.
> > > > Do you have these two PRs for your Celeborn client?
> > >
> > >
> > > No.We mainly use branch-0.5.Would you please consider that merging these two pr into branch-0.5-rc?Those are important features and many users using branch-0.5 may need them.
> >
> >
> > This PR can be helpful in branch-0.5 because branch-0.5 doesn't have PR #2921. Drafting a new RC for branch-0.5 can be a good idea.
>
> Does this PR #2921 also need to be backported to the branch-0.5 branch? @FMX
There are too many conflicts between PR#2921 and branch-0.5. So the PR is not backported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2655805001
> > According to your Jira ticket, "that shuffle fetch fails does not lead to stage fail because task speculation and another attempts succeed", I think the quoted scenario should not happen if you have PR #3080 and #2921.
> > Do you have these two PRs for your Celeborn client?
>
> No.We mainly use branch-0.5.Would you please consider that merging these two pr into branch-0.5-rc?Those are important features and many users using branch-0.5 may need them.
This PR can be helpful in branch-0.5 because branch-0.5 doesn't have PR #2921. Drafting a new RC for branch-0.5 can be a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported [celeborn]
Posted by "gaoyajun02 (via GitHub)" <gi...@apache.org>.
gaoyajun02 commented on PR #3090:
URL: https://github.com/apache/celeborn/pull/3090#issuecomment-2656320345
> > > According to your Jira ticket, "that shuffle fetch fails does not lead to stage fail because task speculation and another attempts succeed", I think the quoted scenario should not happen if you have PR #3080 and #2921.
> > > Do you have these two PRs for your Celeborn client?
> >
> >
> > No.We mainly use branch-0.5.Would you please consider that merging these two pr into branch-0.5-rc?Those are important features and many users using branch-0.5 may need them.
>
> This PR can be helpful in branch-0.5 because branch-0.5 doesn't have PR #2921. Drafting a new RC for branch-0.5 can be a good idea.
Does this PR #2921 also need to be backported to the branch-0.5 branch? @FMX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org