You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sarutak (via GitHub)" <gi...@apache.org> on 2025/09/27 10:18:34 UTC
[PR] [SPARK-53673][CONNECT][TESTS][4.0] Fix a flaky test failure in `SparkSessionE2ESuite - interrupt tag` caused by the usage of `ForkJoinPool` [spark]
sarutak opened a new pull request, #52476:
URL: https://github.com/apache/spark/pull/52476
### What changes were proposed in this pull request?
This PR backports #52417 to `branch-4.0`.
This PR aims to fix one of the issues which block SPARK-48139.
In the problematic test `interrupt tag` in `SparkSessionE2ESuite`, four futures run on threads in `ForkJoinPool` and try to interrupt through tags.
A thread in a `ForkJoinPool` can create a spare thread and make it available in the pool so any of threads can be parent and child. It can happen when a thread performs a blocking operation. One example is `ArrayBlockingQueue.take` and it is called in a method provided by [gRPC](https://github.com/grpc/grpc-java/blob/24085103b926559659ecd3941a3308479876f084/stub/src/main/java/io/grpc/stub/ClientCalls.java#L607).
On the other hand, tags are implemented as [InheritableThreadLocal](https://github.com/apache/spark/blob/13e70100426233e62fd9edf13e229f91f4941ff8/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala#L285).
So, if the futures q1 and q4, or q2 and q3 are parent and child, tags should be inherited, which causes the flaky test failure.
You can easily reproduce the issue by inserting a sleep into the problematic test like as follows (don't forget to replace `ignore` with `test`).
```
// TODO(SPARK-48139): Re-enable `SparkSessionE2ESuite.interrupt tag`
- ignore("interrupt tag") {
+ test("interrupt tag") {
val session = spark
import session.implicits._
@@ -204,6 +204,7 @@ class SparkSessionE2ESuite extends ConnectFunSuite with RemoteSparkSession {
spark.clearTags() // clear for the case of thread reuse by another Future
}
}(executionContext)
+ Thread.sleep(1000)
val q4 = Future {
assert(spark.getTags() == Set())
spark.addTag("one")
```
And then, run the test.
```
$ build/sbt 'connect-client-jvm/testOnly org.apache.spark.sql.connect.SparkSessionE2ESuite -- -z "interrupt tag"'
```
### Why are the changes needed?
For test stability.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Ran the problematic test with inserting sleep like mentioned above and it passed.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53673][CONNECT][TESTS][4.0] Fix a flaky test failure in `SparkSessionE2ESuite - interrupt tag` caused by the usage of `ForkJoinPool` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #52476: [SPARK-53673][CONNECT][TESTS][4.0] Fix a flaky test failure in `SparkSessionE2ESuite - interrupt tag` caused by the usage of `ForkJoinPool`
URL: https://github.com/apache/spark/pull/52476
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53673][CONNECT][TESTS][4.0] Fix a flaky test failure in `SparkSessionE2ESuite - interrupt tag` caused by the usage of `ForkJoinPool` [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #52476:
URL: https://github.com/apache/spark/pull/52476#issuecomment-3356037500
Thank you, @dongjoon-hyun !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53673][CONNECT][TESTS][4.0] Fix a flaky test failure in `SparkSessionE2ESuite - interrupt tag` caused by the usage of `ForkJoinPool` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52476:
URL: https://github.com/apache/spark/pull/52476#issuecomment-3341891303
Merged to branch-4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org