You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "vrozov (via GitHub)" <gi...@apache.org> on 2025/11/14 01:41:01 UTC
[PR] [SPARK-54342][BUILD] Unify scalatest format reports between SBT and maven builds [spark]
vrozov opened a new pull request, #53053:
URL: https://github.com/apache/spark/pull/53053
### What changes were proposed in this pull request?
1. Update SBT scalatest settings to use named file reporter (same as maven scalatest plugin)
2. Update SBT scalatest settings to use the same directories for junit-style xml files as maven scalatest plugin
3. Use without color (W), show all durations (D), show full stack traces (F), show reminder of failed and canceled tests without stack traces (I) for the named file scalatest reporter both in SBT and maven.
### Why are the changes needed?
Unify scalatest settings between SBT and maven. Produce `SparkTestSuite.txt` in the following format:
```
Run starting. Expected test count is: 15
ExecutorSuite:
- SPARK-15963: Catch `TaskKilledException` correctly in Executor.TaskRunner (1 second, 768 milliseconds)
- SPARK-19276: Handle FetchFailedExceptions that are hidden by user exceptions (938 milliseconds)
- Executor's worker threads should be UninterruptibleThread (380 milliseconds)
- SPARK-19276: OOMs correctly handled with a FetchFailure (91 milliseconds)
- SPARK-23816: interrupts are not masked by a FetchFailure (78 milliseconds)
- Gracefully handle error in task deserialization (15 milliseconds)
- Heartbeat should drop zero accumulator updates (248 milliseconds)
- Heartbeat should not drop zero accumulator updates when the conf is disabled (68 milliseconds)
- SPARK-39696: Using accumulators should not cause heartbeat to fail (1 second, 58 milliseconds)
- Send task executor metrics in DirectTaskResult (114 milliseconds)
- Send task executor metrics in TaskKilled (108 milliseconds)
- Send task executor metrics in ExceptionFailure (93 milliseconds)
- SPARK-34949: do not re-register BlockManager when executor is shutting down (67 milliseconds)
- SPARK-33587: isFatalError *** FAILED *** (2 milliseconds)
false equaled false (ExecutorSuite.scala:508)
org.scalatest.exceptions.TestFailedException:
at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
at org.apache.spark.executor.ExecutorSuite.testThrowable$1(ExecutorSuite.scala:508)
at org.apache.spark.executor.ExecutorSuite.$anonfun$new$38(ExecutorSuite.scala:528)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:256)
at org.apache.spark.executor.ExecutorSuite.$anonfun$new$33(ExecutorSuite.scala:527)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:68)
at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:154)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:226)
at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:68)
at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:68)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
at scala.collection.immutable.List.foreach(List.scala:323)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
at org.scalatest.Suite.run(Suite.scala:1114)
at org.scalatest.Suite.run$(Suite.scala:1096)
at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:68)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:68)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
at scala.collection.immutable.List.foreach(List.scala:323)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
at org.scalatest.tools.Runner$.main(Runner.scala:775)
at org.scalatest.tools.Runner.main(Runner.scala)
- SPARK-40235: updateDependencies is interruptible when waiting on lock (18 milliseconds)
Run completed in 6 seconds, 86 milliseconds.
Total number of tests run: 15
Suites: completed 1, aborted 0
Tests: succeeded 14, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***
ExecutorSuite:
- SPARK-33587: isFatalError *** FAILED *** (2 milliseconds)
false equaled false (ExecutorSuite.scala:508)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Run test from maven and sbt
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54342][BUILD] Unify scalatest format reports between SBT and maven builds [spark]
Posted by "vrozov (via GitHub)" <gi...@apache.org>.
vrozov commented on PR #53053:
URL: https://github.com/apache/spark/pull/53053#issuecomment-3564540954
@holdenk @dongjoon-hyun can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54342][BUILD] Unify scalatest format reports between SBT and maven builds [spark]
Posted by "vrozov (via GitHub)" <gi...@apache.org>.
vrozov commented on PR #53053:
URL: https://github.com/apache/spark/pull/53053#issuecomment-3721469854
@dongjoon-hyun can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54342][BUILD] Unify scalatest format reports between SBT and maven builds [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #53053: [SPARK-54342][BUILD] Unify scalatest format reports between SBT and maven builds
URL: https://github.com/apache/spark/pull/53053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54342][BUILD] Unify scalatest format reports between SBT and maven builds [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53053:
URL: https://github.com/apache/spark/pull/53053#issuecomment-4274881268
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54342][BUILD] Unify scalatest format reports between SBT and maven builds [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #53053:
URL: https://github.com/apache/spark/pull/53053#issuecomment-3564549741
Thanks for pinging me. I’m a bit low on bandwidth right now because of the ongoing Apache Spark 4.1.0 release, but I can revisit this after the release. @vrozov .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org