You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2021/01/06 03:18:00 UTC
[jira] [Resolved] (SPARK-33029) Standalone mode blacklist executors
page UI marks driver as blacklisted
[ https://issues.apache.org/jira/browse/SPARK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-33029.
-----------------------------------
Fix Version/s: 3.1.0
Resolution: Fixed
Issue resolved by pull request 30954
[https://github.com/apache/spark/pull/30954]
> Standalone mode blacklist executors page UI marks driver as blacklisted
> -----------------------------------------------------------------------
>
> Key: SPARK-33029
> URL: https://issues.apache.org/jira/browse/SPARK-33029
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Assignee: Baohe Zhang
> Priority: Major
> Fix For: 3.1.0
>
> Attachments: Screen Shot 2020-09-29 at 1.52.09 PM.png, Screen Shot 2020-09-29 at 1.53.37 PM.png
>
>
> I am running a spark shell on a 1 node standalone cluster. I noticed that the executors page ui was marking the driver as blacklisted for the stage that is running. Attached a screen shot.
> Also, in my case one of the executors died and it doesn't seem like the schedule rpicked up the new one. It doesn't show up on the stages page and just shows it as active but none of the tasks ran there.
>
> You can reproduce this by starting a master and slave on a single node, then launch a shell like where you will get multiple executors (in this case I got 3)
> $SPARK_HOME/bin/spark-shell --master spark://yourhost:7077 --executor-cores 4 --conf spark.blacklist.enabled=true
>
> From shell run:
> {code:java}
> import org.apache.spark.TaskContext
> val rdd = sc.makeRDD(1 to 1000, 5).mapPartitions { it =>
> val context = TaskContext.get()
> if (context.attemptNumber() < 2) {
> throw new Exception("test attempt num")
> }
> it
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org