You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/02/06 18:27:39 UTC
[jira] [Closed] (SPARK-13218) Executor failed after SparkContext
start and start again
[ https://issues.apache.org/jira/browse/SPARK-13218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen closed SPARK-13218.
-----------------------------
> Executor failed after SparkContext start and start again
> ----------------------------------------------------------
>
> Key: SPARK-13218
> URL: https://issues.apache.org/jira/browse/SPARK-13218
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.6.0
> Environment: Run IPython/Jupyter along with Spark on ubuntu 14.04
> Reporter: leo wu
>
> In a python notebook, I am trying to stop SparkContext which is initialized with local master and then start again with conf over a remote Spark standalone cluster, like :
> import sys
> from random import random
> import atexit
> import os
> import platform
> import py4j
> import pyspark
> from pyspark import SparkContext, SparkConf
> from pyspark.sql import SQLContext, HiveContext
> from pyspark.storagelevel import StorageLevel
> os.environ["SPARK_HOME"] = "/home/notebook/spark-1.6.0-bin-hadoop2.6"
> os.environ["PYSPARK_SUBMIT_ARGS"] = "--master spark://10.115.89.219:7077"
> os.environ["SPARK_LOCAL_HOSTNAME"] = "wzymaster2011"
> SparkContext.setSystemProperty("spark.master", "spark://10.115.89.219:7077")
> SparkContext.setSystemProperty("spark.cores.max", "4")
> SparkContext.setSystemProperty("spark.driver.host", "wzymaster2011")
> SparkContext.setSystemProperty("spark.driver.port", "9000")
> SparkContext.setSystemProperty("spark.blockManager.port", "9001")
> SparkContext.setSystemProperty("spark.fileserver.port", "9002")
> conf = SparkConf().setAppName("Python-Test")
> sc = SparkContext(conf=conf)
> However, I always get error in Executor like :
> 16/02/05 14:37:32 DEBUG BlockManager: Getting remote block broadcast_0_piece0 from BlockManagerId(driver, localhost, 9002)
> 16/02/05 14:37:32 DEBUG TransportClientFactory: Creating new connection to localhost/127.0.0.1:9002
> 16/02/05 14:37:32 ERROR RetryingBlockFetcher: Exception while beginning fetch of 1 outstanding blocks
> java.io.IOException: Failed to connect to localhost/127.0.0.1:9002
> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
> at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:90)
> at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
> I suspect that new SparkConf isn't properly passed to executor through Spark Master for some reason.
> Please advise it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org