You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2020/05/18 05:57:18 UTC
[spark] branch branch-3.0 updated (cbd8568 -> 2cdf4eb)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git.
from cbd8568 [SPARK-31746][YARN][TESTS] Show the actual error message in LocalityPlacementStrategySuite
new 88e00c3 Revert "[SPARK-31746][YARN][TESTS] Show the actual error message in LocalityPlacementStrategySuite"
new 2cdf4eb [SPARK-31746][YARN][TESTS] Show the actual error message in LocalityPlacementStrategySuite
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/spark/deploy/yarn/LocalityPlacementStrategySuite.scala | 1 +
1 file changed, 1 insertion(+)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 02/02: [SPARK-31746][YARN][TESTS] Show the actual error
message in LocalityPlacementStrategySuite
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 2cdf4eb7c20393e47709d02454356981df05040a
Author: HyukjinKwon <gu...@apache.org>
AuthorDate: Mon May 18 14:35:02 2020 +0900
[SPARK-31746][YARN][TESTS] Show the actual error message in LocalityPlacementStrategySuite
This PR proposes to show the actual traceback when "handle large number of containers and tasks (SPARK-18750)" test fails in `LocalityPlacementStrategySuite`.
**It does not fully resolve the JIRA SPARK-31746 yet**. I tried to reproduce in my local by controlling the factors in the tests but I couldn't. I double checked the changes in SPARK-18750 are still valid.
This test is flaky for an unknown reason (see https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122768/testReport/org.apache.spark.deploy.yarn/LocalityPlacementStrategySuite/handle_large_number_of_containers_and_tasks__SPARK_18750_/):
```
sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError did not equal null
at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:503)
```
After this PR, it will help to investigate the root cause:
**Before**:
```
[info] - handle large number of containers and tasks (SPARK-18750) *** FAILED *** (824 milliseconds)
[info] java.lang.StackOverflowError did not equal null (LocalityPlacementStrategySuite.scala:49)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
[info] at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
[info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:503)
[info] at org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:49)
[info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
[info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:157)
[info] at org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
[info] at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:286)
[info] at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
[info] at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
...
```
**After**:
```
[info] - handle large number of containers and tasks (SPARK-18750) *** FAILED *** (825 milliseconds)
[info] StackOverflowError should not be thrown; however, got:
[info]
[info] java.lang.StackOverflowError
[info] at scala.collection.TraversableLike.$anonfun$filterImpl$1(TraversableLike.scala:256)
[info] at scala.collection.MapLike$MappedValues.$anonfun$foreach$3(MapLike.scala:256)
[info] at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
[info] at scala.collection.MapLike$MappedValues.$anonfun$foreach$3(MapLike.scala:256)
[info] at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
[info] at scala.collection.MapLike$MappedValues.$anonfun$foreach$3(MapLike.scala:256)
[info] at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
[info] at scala.collection.MapLike$MappedValues.$anonfun$foreach$3(MapLike.scala:256)
[info] at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
[info] at scala.collection.MapLike$MappedValues.$anonfun$foreach$3(MapLike.scala:256)
[info] at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
[info] at scala.collection.MapLike$MappedValues.$anonfun$foreach$3(MapLike.scala:256)
[info] at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
[info] at scala.collection.MapLike$MappedValues.$anonfun$foreach$3(MapLike.scala:256)
[info] at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
[info] at scala.collection.MapLike$MappedValues.$anonfun$foreach$3(MapLike.scala:256)
...
```
No, dev-only.
Manually tested by reverting https://github.com/apache/spark/commit/76db394f2baedc2c7b7a52c05314a64ec9068263 locally.
Closes #28566 from HyukjinKwon/SPARK-31746.
Authored-by: HyukjinKwon <gu...@apache.org>
Signed-off-by: HyukjinKwon <gu...@apache.org>
(cherry picked from commit 3bf7bf99e96fab754679a4f3c893995263161341)
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
.../spark/deploy/yarn/LocalityPlacementStrategySuite.scala | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/LocalityPlacementStrategySuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/LocalityPlacementStrategySuite.scala
index b7f2565..bb14e09 100644
--- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/LocalityPlacementStrategySuite.scala
+++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/LocalityPlacementStrategySuite.scala
@@ -17,7 +17,8 @@
package org.apache.spark.deploy.yarn
-import scala.collection.JavaConverters._
+import java.io.{PrintWriter, StringWriter}
+
import scala.collection.mutable.{HashMap, HashSet, Set}
import org.apache.hadoop.yarn.api.records._
@@ -45,7 +46,11 @@ class LocalityPlacementStrategySuite extends SparkFunSuite {
thread.start()
thread.join()
- assert(error === null)
+ if (error != null) {
+ val errors = new StringWriter()
+ error.printStackTrace(new PrintWriter(errors))
+ fail(s"StackOverflowError should not be thrown; however, got:\n\n$errors")
+ }
}
private def runTest(): Unit = {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 01/02: Revert "[SPARK-31746][YARN][TESTS] Show the actual
error message in LocalityPlacementStrategySuite"
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 88e00c367bccf51a343abae2ef734fa028c46f7b
Author: HyukjinKwon <gu...@apache.org>
AuthorDate: Mon May 18 14:55:31 2020 +0900
Revert "[SPARK-31746][YARN][TESTS] Show the actual error message in LocalityPlacementStrategySuite"
This reverts commit cbd8568ad7588cf14e5519c5aabf88d0b7fb0e33.
---
.../spark/deploy/yarn/LocalityPlacementStrategySuite.scala | 10 +++-------
1 file changed, 3 insertions(+), 7 deletions(-)
diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/LocalityPlacementStrategySuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/LocalityPlacementStrategySuite.scala
index 3c77872..b7f2565 100644
--- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/LocalityPlacementStrategySuite.scala
+++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/LocalityPlacementStrategySuite.scala
@@ -17,8 +17,7 @@
package org.apache.spark.deploy.yarn
-import java.io.{PrintWriter, StringWriter}
-
+import scala.collection.JavaConverters._
import scala.collection.mutable.{HashMap, HashSet, Set}
import org.apache.hadoop.yarn.api.records._
@@ -46,11 +45,7 @@ class LocalityPlacementStrategySuite extends SparkFunSuite {
thread.start()
thread.join()
- if (error != null) {
- val errors = new StringWriter()
- error.printStackTrace(new PrintWriter(errors))
- fail(s"StackOverflowError should not be thrown; however, got:\n\n$errors")
- }
+ assert(error === null)
}
private def runTest(): Unit = {
@@ -61,6 +56,7 @@ class LocalityPlacementStrategySuite extends SparkFunSuite {
// goal is to create enough requests for localized containers (so there should be many
// tasks on several hosts that have no allocated containers).
+ val resource = Resource.newInstance(8 * 1024, 4)
val strategy = new LocalityPreferredContainerPlacementStrategy(new SparkConf(),
yarnConf, resource, new MockResolver())
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org