You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/16 10:38:50 UTC

[GitHub] [spark] sarutak opened a new pull request #34922: [SPARK-37663][CORE][TESTS] Mitigate ConcurrentModificationException thrown from tests in SparkContextSuite

sarutak opened a new pull request #34922:
URL: https://github.com/apache/spark/pull/34922

### What changes were proposed in this pull request?

This PR fixes an issue that some tests in `SparkContextSuite` can throw `ConcurrentModificationException` with Scala 2.13.
https://github.com/apache/spark/runs/4543047740?check_suite_focus=true#step:9:20851
The cause seems to be same as SPARK-37315 (#34583).

You can easily reproduce this issue by applying the following diff to `master`.
```
diff --git a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
index bc809f11cc..e5dde84c6e 100644
--- a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
@@ -1130,9 +1130,11 @@ class SparkContextSuite extends SparkFunSuite with LocalSparkContext with Eventu
sc.addJar("ivy://org.apache.hive:hive-storage-api:2.7.0?" +
"invalidParam1=foo&invalidParam2=boo")
assert(sc.listJars().exists(_.contains("org.apache.hive_hive-storage-api-2.7.0.jar")))
- assert(logAppender.loggingEvents.exists(_.getRenderedMessage.contains(
- "Invalid parameters `invalidParam1,invalidParam2` found in Ivy URI query " +
- "`invalidParam1=foo&invalidParam2=boo`.")))
+ assert(logAppender.loggingEvents.exists { x =>
+ Thread.sleep(1000)
+ x.getRenderedMessage.contains(
+ "Invalid parameters `invalidParam1,invalidParam2` found in Ivy URI query " +
+ "`invalidParam1=foo&invalidParam2=boo`.")})
}
}
```

### Why are the changes needed?

Fix the flaky test.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Locally checked that no `ConcurrentModificationException` is thrown even if a sleep is inserted like the diff shown above.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org