You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrei Iatsuk (JIRA)" <ji...@apache.org> on 2018/06/27 15:53:00 UTC
[jira] [Issue Comment Deleted] (SPARK-12009) Avoid re-allocate yarn
container while driver want to stop all Executors
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Iatsuk updated SPARK-12009:
----------------------------------
Comment: was deleted
(was: Apache Spark 2.1.2:
{code:java}
SparkSession sparkSession = SparkSession.builder().appName("Test").master("local[*]").getOrCreate();
sparkSession.createDataset(Arrays.asList(1, 2, 3, 4, 5, 6), Encoders.INT()).count();
sparkSession.close();
sparkSession = SparkSession.builder().appName("Test").master("local[*]").getOrCreate();
sparkSession.createDataset(Arrays.asList(1, 2, 3, 4, 5, 6), Encoders.INT()).count(); // this throws Exception
{code}
Execption message:
{noformat}
[LiveListenerBus] SparkListenerBus has already stopped! Dropping event SparkListenerSQLExecutionStart(1,count at SparkTest.java:15,org.apache.spark.sql.Dataset.count(Dataset.scala:2419)
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree: ...
Caused by: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
{noformat}
)
> Avoid re-allocate yarn container while driver want to stop all Executors
> ------------------------------------------------------------------------
>
> Key: SPARK-12009
> URL: https://issues.apache.org/jira/browse/SPARK-12009
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.5.2
> Reporter: SuYan
> Assignee: SuYan
> Priority: Minor
> Fix For: 2.0.0
>
>
> Log based 1.4.0
> 2015-11-26,03:05:16,176 WARN org.spark-project.jetty.util.thread.QueuedThreadPool: 8 threads could not be stopped
> 2015-11-26,03:05:16,177 INFO org.apache.spark.ui.SparkUI: Stopped Spark web UI at http://
> 2015-11-26,03:05:16,401 INFO org.apache.spark.scheduler.DAGScheduler: Stopping DAGScheduler
> 2015-11-26,03:05:16,450 INFO org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Shutting down all executors
> 2015-11-26,03:05:16,525 INFO org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
> 2015-11-26,03:05:16,791 INFO org.apache.spark.deploy.yarn.ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. XX.XX.XX.XX:38734
> 2015-11-26,03:05:16,847 ERROR org.apache.spark.scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(164,WrappedArray())
> 2015-11-26,03:05:27,242 INFO org.apache.spark.deploy.yarn.YarnAllocator: Will request 13 executor containers, each with 1 cores and 4608 MB memory including 1024 MB overhead
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org