You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "Z1Wu (via GitHub)" <gi...@apache.org> on 2025/05/07 07:34:50 UTC
[PR] [CELEBORN-1601][FOLLOWUP] Remove redundant logs of expired shuffle in master-worker heartbeat [celeborn]
Z1Wu opened a new pull request, #3244:
URL: https://github.com/apache/celeborn/pull/3244
### What changes were proposed in this pull request?
remove redundant logs of expired shuffle in master-worker heartbeat
### Why are the changes needed?
- expired key already log in debug level
https://github.com/apache/celeborn/blob/88661c2c6934ff8695c85408cc02519060b534ef/master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala#L702-L703
- Expired shuffle logs is too noisy. And it seems like a common and normal behavior, shouldn't be logged in WARN-level.


### Does this PR introduce _any_ user-facing change?
yes, master log will be much less.
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1601][FOLLOWUP] Remove redundant logs of expired shuffle in master-worker heartbeat [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #3244:
URL: https://github.com/apache/celeborn/pull/3244#discussion_r2079328499
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -694,8 +694,6 @@ private[celeborn] class Master(
val (appId, shuffleId) = Utils.splitShuffleKey(shuffleKey)
val shuffleIds = statusSystem.registeredAppAndShuffles.get(appId)
if (shuffleIds == null || !shuffleIds.contains(shuffleId)) {
- logWarning(
Review Comment:
The log can be very useful for troubleshooting, IMO better to keep it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1601][FOLLOWUP] Remove redundant logs of expired shuffle in master-worker heartbeat [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3244:
URL: https://github.com/apache/celeborn/pull/3244#discussion_r2094542641
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -694,8 +694,6 @@ private[celeborn] class Master(
val (appId, shuffleId) = Utils.splitShuffleKey(shuffleKey)
val shuffleIds = statusSystem.registeredAppAndShuffles.get(appId)
if (shuffleIds == null || !shuffleIds.contains(shuffleId)) {
- logWarning(
- s"Shuffle $shuffleKey expired on $host:$rpcPort:$pushPort:$fetchPort:$replicatePort.")
Review Comment:
You can define a class named `ShuffleAuditLogger` and log the expired info in separate log file.
Similar with https://github.com/apache/celeborn/pull/2895
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1601][FOLLOWUP] Remove redundant logs of expired shuffle in master-worker heartbeat [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3244:
URL: https://github.com/apache/celeborn/pull/3244#discussion_r2094566433
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -694,8 +694,6 @@ private[celeborn] class Master(
val (appId, shuffleId) = Utils.splitShuffleKey(shuffleKey)
val shuffleIds = statusSystem.registeredAppAndShuffles.get(appId)
if (shuffleIds == null || !shuffleIds.contains(shuffleId)) {
- logWarning(
- s"Shuffle $shuffleKey expired on $host:$rpcPort:$pushPort:$fetchPort:$replicatePort.")
Review Comment:
Raised https://github.com/apache/celeborn/pull/3265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1601][FOLLOWUP] Remove redundant logs of expired shuffle in master-worker heartbeat [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei closed pull request #3244: [CELEBORN-1601][FOLLOWUP] Remove redundant logs of expired shuffle in master-worker heartbeat
URL: https://github.com/apache/celeborn/pull/3244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org