You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/09/25 02:03:04 UTC
[jira] [Assigned] (SPARK-10812) Spark Hadoop Util does not support
stopping a non-yarn Spark Context & starting a Yarn spark context.
[ https://issues.apache.org/jira/browse/SPARK-10812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-10812:
------------------------------------
Assignee: Apache Spark
> Spark Hadoop Util does not support stopping a non-yarn Spark Context & starting a Yarn spark context.
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-10812
> URL: https://issues.apache.org/jira/browse/SPARK-10812
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Reporter: holdenk
> Assignee: Apache Spark
> Priority: Minor
>
> While this is likely not a huge issue for real production systems, for test systems which may setup a Spark Context and tear it down and stand up a Spark Context with a different master (e.g. some local mode & some yarn mode) tests this cane be an issue. Discovered during work on spark-testing-base on Spark 1.4.1, but seems like the logic that triggers it is present in master (see SparkHadoopUtil object). A valid work around for users encountering this issue is to fork a different JVM, however this can be heavy weight.
> {quote}
> [info] SampleMiniClusterTest:
> [info] Exception encountered when attempting to run a suite with class name: com.holdenkarau.spark.testing.SampleMiniClusterTest *** ABORTED ***
> [info] java.lang.ClassCastException: org.apache.spark.deploy.SparkHadoopUtil cannot be cast to org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
> [info] at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:163)
> [info] at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:257)
> [info] at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:561)
> [info] at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:115)
> [info] at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
> [info] at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
> [info] at org.apache.spark.SparkContext.<init>(SparkContext.scala:497)
> [info] at com.holdenkarau.spark.testing.SharedMiniCluster$class.setup(SharedMiniCluster.scala:186)
> [info] at com.holdenkarau.spark.testing.SampleMiniClusterTest.setup(SampleMiniClusterTest.scala:26)
> [info] at com.holdenkarau.spark.testing.SharedMiniCluster$class.beforeAll(SharedMiniCluster.scala:103)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org