You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2023/01/17 23:45:58 UTC
[GitHub] [spark] akpatnam25 opened a new pull request, #39632: Spark 41415 spark 42090 backport
akpatnam25 opened a new pull request, #39632:
URL: https://github.com/apache/spark/pull/39632
### What changes were proposed in this pull request?
Add the ability to retry SASL requests. Will add it as a metric too soon to track SASL retries.
### Why are the changes needed?
We are seeing increased SASL timeouts internally, and this issue would mitigate the issue. We already have this feature enabled for our 2.3 jobs, and we have seen failures significantly decrease.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit tests, and tested on cluster to ensure the retries are being triggered correctly.
Closes https://github.com/apache/spark/pull/38959 from akpatnam25/[SPARK-41415](https://issues.apache.org/jira/browse/SPARK-41415).
Authored-by: Aravind Patnam <ap...@linkedin.com>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
=================================================
### What changes were proposed in this pull request?
This PR introduces sasl retry count in RetryingBlockTransferor.
### Why are the changes needed?
Previously a boolean variable, saslTimeoutSeen, was used. However, the boolean variable wouldn't cover the following scenario:
1. SaslTimeoutException
2. IOException
3. SaslTimeoutException
4. IOException
Even though IOException at https://github.com/apache/spark/pull/2 is retried (resulting in increment of retryCount), the retryCount would be cleared at step https://github.com/apache/spark/pull/4.
Since the intention of saslTimeoutSeen is to undo the increment due to retrying SaslTimeoutException, we should keep a counter for SaslTimeoutException retries and subtract the value of this counter from retryCount.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
New test is added, courtesy of Mridul.
Closes https://github.com/apache/spark/pull/39611 from tedyu/sasl-cnt.
Authored-by: Ted Yu <yu...@gmail.com>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #39632: SPARK-41415/SPARK-42090 Backport to 3.2
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on PR #39632:
URL: https://github.com/apache/spark/pull/39632#issuecomment-1386252421
@dongjoon-hyun oops tagged the wrong person :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 closed pull request #39632: SPARK-41415/SPARK-42090 Backport to 3.2
Posted by GitBox <gi...@apache.org>.
akpatnam25 closed pull request #39632: SPARK-41415/SPARK-42090 Backport to 3.2
URL: https://github.com/apache/spark/pull/39632
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #39632: SPARK-41415/SPARK-42090 Backport to 3.2
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on PR #39632:
URL: https://github.com/apache/spark/pull/39632#issuecomment-1386249063
@mridulm @Dooyoung-Hwang @tedyu backport into 3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #39632: SPARK-41415/SPARK-42090 Backport to 3.2
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on PR #39632:
URL: https://github.com/apache/spark/pull/39632#issuecomment-1386249238
+ CC @otterc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org