You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "s0nskar (via GitHub)" <gi...@apache.org> on 2026/05/06 09:18:51 UTC
[PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
s0nskar opened a new pull request, #3675:
URL: https://github.com/apache/celeborn/pull/3675
…upt shuffle
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Fix backward compatibility issue with interrupt shuffle by checking if the reason is nonEmpty.
### Why are the changes needed?
If someone uses a new client with old server which is not sending `CheckQuotaResponse` in `HeartbeatFromApplicationResponse` then proto uses default value to build CheckQuotaResponse with isAvailable=false and reason="". In this case the job will always fails without breaching the quota, we should not fail the job if the reason is empty to make it backward compatible.
### Does this PR resolve a correctness bug?
<!-- Yes/No. (Note: If yes, committer will add `correctness` label to current pull request). -->
No
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Tested in local setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3675:
URL: https://github.com/apache/celeborn/pull/3675#discussion_r3223337803
##########
client/src/main/scala/org/apache/celeborn/client/ApplicationHeartbeater.scala:
##########
@@ -164,7 +164,7 @@ class ApplicationHeartbeater(
}
private def checkQuotaExceeds(response: CheckQuotaResponse): Unit = {
- if (conf.quotaInterruptShuffleEnabled && !response.isAvailable) {
+ if (conf.quotaInterruptShuffleEnabled && !response.isAvailable && response.reason.nonEmpty) {
Review Comment:
It's better to introduce new field like `quotaCheckSupported` to fix backward compatiblity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3675:
URL: https://github.com/apache/celeborn/pull/3675#issuecomment-4627636899
Thanks. Merged to main(v0.7.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas closed pull request #3675: [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle
URL: https://github.com/apache/celeborn/pull/3675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #3675:
URL: https://github.com/apache/celeborn/pull/3675#issuecomment-4394158546
Using `reason.nonEmpty` as a proxy for "server supports quota check" is fragile — `reason` is a business description field, not a compatibility signal. If a legitimate quota-exceeded response happens to have an empty reason, it would be silently ignored; conversely, any future change to the reason format could break this logic.
Since `quotaInterruptShuffleEnabled` defaults to `false`, users must explicitly opt in. In a new-client/old-server scenario, users should simply not enable this config. This is an explicit and semantically clear approach, rather than relying on an implicit convention around the `reason` field.
If we do need robust backward compatibility, a better approach would be to add an explicit capability field (e.g., `quotaCheckSupported`) to the heartbeat response. But that's a larger change and can be addressed separately.
— *Reviewed by [Claude Code](https://claude.com/claude-code)*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3675:
URL: https://github.com/apache/celeborn/pull/3675#issuecomment-4532047026
@s0nskar, could you please help to update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3675:
URL: https://github.com/apache/celeborn/pull/3675#issuecomment-4490776134
@s0nskar, please rebase the latest main branch for Grafana Dashboard CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3675:
URL: https://github.com/apache/celeborn/pull/3675#issuecomment-4602697177
@SteNicholas / @RexXiong PTAL. Instead of adding a new field, i have made changes in the parsing of HeartbeatFromApplicationResponse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3675:
URL: https://github.com/apache/celeborn/pull/3675#issuecomment-4602116285
@s0nskar, any update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3675:
URL: https://github.com/apache/celeborn/pull/3675#issuecomment-4563595090
Ack, will work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3675:
URL: https://github.com/apache/celeborn/pull/3675#issuecomment-4426853853
@s0nskar, please rebase the latest main branch for Grafana Dashboard CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on code in PR #3675:
URL: https://github.com/apache/celeborn/pull/3675#discussion_r3342143939
##########
client/src/main/scala/org/apache/celeborn/client/ApplicationHeartbeater.scala:
##########
@@ -164,7 +164,7 @@ class ApplicationHeartbeater(
}
private def checkQuotaExceeds(response: CheckQuotaResponse): Unit = {
- if (conf.quotaInterruptShuffleEnabled && !response.isAvailable) {
+ if (conf.quotaInterruptShuffleEnabled && !response.isAvailable && response.reason.nonEmpty) {
Review Comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org