You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/03/30 02:59:00 UTC
[jira] [Updated] (SPARK-34900) Some `spark-submit` commands used to run benchmarks in the user's guide is wrong
[ https://issues.apache.org/jira/browse/SPARK-34900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-34900:
---------------------------------
Fix Version/s: 3.2.0
> Some `spark-submit` commands used to run benchmarks in the user's guide is wrong
> ---------------------------------------------------------------------------------
>
> Key: SPARK-34900
> URL: https://issues.apache.org/jira/browse/SPARK-34900
> Project: Spark
> Issue Type: Bug
> Components: Tests
> Affects Versions: 3.2.0
> Reporter: Yang Jie
> Priority: Trivial
> Fix For: 3.2.0
>
>
> For example, the guide for running JoinBenchmark as follows:
>
> {code:java}
> /**
> * Benchmark to measure performance for joins.
> * To run this benchmark:
> * {{{
> * 1. without sbt:
> * bin/spark-submit --class <this class> --jars <spark core test jar> <spark sql test jar>
> * 2. build/sbt "sql/test:runMain <this class>"
> * 3. generate result:
> * SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"
> * Results will be written to "benchmarks/JoinBenchmark-results.txt".
> * }}}
> */
> object JoinBenchmark extends SqlBasedBenchmark {
> {code}
>
>
> but if we run JoinBenchmark with commnad
>
> {code:java}
> bin/spark-submit --class org.apache.spark.sql.execution.benchmark.JoinBenchmark --jars spark-core_2.12-3.2.0-SNAPSHOT-tests.jar spark-sql_2.12-3.2.0-SNAPSHOT-tests.jar
> {code}
>
> The following exception will be thrown：
>
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/plans/SQLHelper
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369){code}
>
> because SqlBasedBenchmark trait extends BenchmarkBase and SQLHelper, SQLHelper def in spark-catalyst-tests.jar.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org