You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2020/12/28 19:39:00 UTC
[jira] [Assigned] (SPARK-33029) Standalone mode blacklist executors
page UI marks driver as blacklisted
[ https://issues.apache.org/jira/browse/SPARK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-33029:
------------------------------------
Assignee: (was: Apache Spark)
> Standalone mode blacklist executors page UI marks driver as blacklisted
> -----------------------------------------------------------------------
>
> Key: SPARK-33029
> URL: https://issues.apache.org/jira/browse/SPARK-33029
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Priority: Major
> Attachments: Screen Shot 2020-09-29 at 1.52.09 PM.png, Screen Shot 2020-09-29 at 1.53.37 PM.png
>
>
> I am running a spark shell on a 1 node standalone cluster. I noticed that the executors page ui was marking the driver as blacklisted for the stage that is running. Attached a screen shot.
> Also, in my case one of the executors died and it doesn't seem like the schedule rpicked up the new one. It doesn't show up on the stages page and just shows it as active but none of the tasks ran there.
>
> You can reproduce this by starting a master and slave on a single node, then launch a shell like where you will get multiple executors (in this case I got 3)
> $SPARK_HOME/bin/spark-shell --master spark://yourhost:7077 --executor-cores 4 --conf spark.blacklist.enabled=true
>
> From shell run:
> {code:java}
> import org.apache.spark.TaskContext
> val rdd = sc.makeRDD(1 to 1000, 5).mapPartitions { it =>
> val context = TaskContext.get()
> if (context.attemptNumber() < 2) {
> throw new Exception("test attempt num")
> }
> it
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org