You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/09/16 04:51:00 UTC
[jira] [Commented] (SPARK-36774) Use SparkSubmitTestUtils to core
and use it in SparkSubmitSuite
[ https://issues.apache.org/jira/browse/SPARK-36774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415869#comment-17415869 ]
Apache Spark commented on SPARK-36774:
--------------------------------------
User 'JoshRosen' has created a pull request for this issue:
https://github.com/apache/spark/pull/34013
> Use SparkSubmitTestUtils to core and use it in SparkSubmitSuite
> ---------------------------------------------------------------
>
> Key: SPARK-36774
> URL: https://issues.apache.org/jira/browse/SPARK-36774
> Project: Spark
> Issue Type: Improvement
> Components: Tests
> Affects Versions: 3.0.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Minor
>
> {{SparkSubmitSuite}} test failures can be painful to debug: currently, failed tests result in messages like
> {code:java}
> org.scalatest.exceptions.TestFailedException: Process returned with exit code 1. See the log4j logs for more detail.
> {code}
> which require the developer to hunt in test logs in order to identify the root cause of the test failure.
> In contrast, {{HiveSparkSubmitSuite}}'s output packs lots more useful information into its test failure messages:
> {code:java}
> [info] - temporary Hive UDF: define a UDF and use it *** FAILED *** (4 seconds, 135 milliseconds)
> [info] spark-submit returned with exit code 101.
> [info] Command line: '/Users/joshrosen/oss-spark/bin/spark-submit' '--class' 'wrongClassName' '--name' 'TemporaryHiveUDFTest' '--master' 'local-cluster[2,1,1024]' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--driver-java-options' '-Dderby.system.durability=test' '--jars' 'file:/Users/joshrosen/oss-spark/target/tmp/spark-32d0a47c-33eb-4488-866a-9994b3727b5b/testJar-1631766233448.jar,file:/Users/joshrosen/oss-spark/target/tmp/spark-e9e32588-83fa-43bd-a60f-9538fd30ab9a/testJar-1631766233601.jar' 'file:/Users/joshrosen/oss-spark/target/tmp/spark-daef4628-d953-41e3-b935-a726a7796418/testJar-1631766232701.jar' 'SparkSubmitClassA' 'SparkSubmitClassB'
> [info]
> [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Class path contains multiple SLF4J bindings.
> [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Found binding in [jar:file:/Users/joshrosen/oss-spark/assembly/target/scala-2.12/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Found binding in [jar:file:/Users/joshrosen/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> [info] 2021-09-15 21:23:54.753 - stderr> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [info] 2021-09-15 21:23:55.623 - stderr> Error: Failed to load class wrongClassName. (SparkSubmitTestUtils.scala:97)
> [info] org.scalatest.exceptions.TestFailedException:
> [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
> [...]
> {code}
> This is done via code in {{SparkSubmitTestUtils}}, which is currently located in the {{sql/hive}} module. I propose a refactoring to move this into the {{core}} module so that it can be used by both {{SparkSubmitSuite}} and {{HiveSparkSubmitSuite}}, enriching the former with nicer error outputs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org