You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/02/17 02:04:11 UTC
[jira] [Resolved] (SPARK-1670) PySpark Fails to Create SparkContext
Due To Debugging Options in conf/java-opts
[ https://issues.apache.org/jira/browse/SPARK-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen resolved SPARK-1670.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.3.0
The root cause, SPARK-2313, was fixed for 1.3, so I'm going to mark this as Fixed.
> PySpark Fails to Create SparkContext Due To Debugging Options in conf/java-opts
> -------------------------------------------------------------------------------
>
> Key: SPARK-1670
> URL: https://issues.apache.org/jira/browse/SPARK-1670
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.0.0
> Environment: pats-air:spark pat$ IPYTHON=1 bin/pyspark
> Python 2.7.5 (default, Aug 25 2013, 00:04:04)
> ...
> IPython 1.1.0
> ...
> Spark version 1.0.0-SNAPSHOT
> Using Python version 2.7.5 (default, Aug 25 2013 00:04:04)
> Reporter: Pat McDonough
> Fix For: 1.3.0
>
>
> When JVM debugging options are in conf/java-opts, it causes pyspark to fail when creating the SparkContext. The java-opts file looks like the following:
> {code}-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
> {code}
> Here's the error:
> {code}---------------------------------------------------------------------------
> ValueError Traceback (most recent call last)
> /Library/Python/2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
> 202 else:
> 203 filename = fname
> --> 204 __builtin__.execfile(filename, *where)
> /Users/pat/Projects/spark/python/pyspark/shell.py in <module>()
> 41 SparkContext.setSystemProperty("spark.executor.uri", os.environ["SPARK_EXECUTOR_URI"])
> 42
> ---> 43 sc = SparkContext(os.environ.get("MASTER", "local[*]"), "PySparkShell", pyFiles=add_files)
> 44
> 45 print("""Welcome to
> /Users/pat/Projects/spark/python/pyspark/context.pyc in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway)
> 92 tempNamedTuple = namedtuple("Callsite", "function file linenum")
> 93 self._callsite = tempNamedTuple(function=None, file=None, linenum=None)
> ---> 94 SparkContext._ensure_initialized(self, gateway=gateway)
> 95
> 96 self.environment = environment or {}
> /Users/pat/Projects/spark/python/pyspark/context.pyc in _ensure_initialized(cls, instance, gateway)
> 172 with SparkContext._lock:
> 173 if not SparkContext._gateway:
> --> 174 SparkContext._gateway = gateway or launch_gateway()
> 175 SparkContext._jvm = SparkContext._gateway.jvm
> 176 SparkContext._writeToFile = SparkContext._jvm.PythonRDD.writeToFile
> /Users/pat/Projects/spark/python/pyspark/java_gateway.pyc in launch_gateway()
> 44 proc = Popen(command, stdout=PIPE, stdin=PIPE)
> 45 # Determine which ephemeral port the server started on:
> ---> 46 port = int(proc.stdout.readline())
> 47 # Create a thread to echo output from the GatewayServer, which is required
> 48 # for Java log output to show up:
> ValueError: invalid literal for int() with base 10: 'Listening for transport dt_socket at address: 5005\n'
> {code}
> Note that when you use JVM debugging, the very first line of output (e.g. when running spark-shell) looks like this:
> {code}Listening for transport dt_socket at address: 5005{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org