[jira] [Created] (SPARK-1904) ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

["Eric Lee (JIRA)" <ji...@apache.org>]

Eric Lee created SPARK-1904:
-------------------------------

             Summary: ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark
                 Key: SPARK-1904
                 URL: https://issues.apache.org/jira/browse/SPARK-1904
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 1.0.0
         Environment: Ubuntu AMI in EC2, build taken from tags/spark-v1.0.0 tag

$ lsb_release -r
Release:	12.04
$ uname -a
Linux ip-10-97-159-136 3.2.0-23-virtual #36-Ubuntu SMP Tue Apr 10 22:29:03 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$
            Reporter: Eric Lee


Attempting to run my bin/pyspark interactive shell against a Mesos cluster this assignment to MASTER used to work:
     MASTER=zk://10.32.43.173:2181,10.33.9.56:2181,10.116.167.95:2181/mes

Now, with this spark-v1.0.0 tag, this yields an error:

Type "help", "copyright", "credits" or "license" for more information.
command is ['/tmp/spark-1.0.0-rc10/./bin/spark-submit', 'pyspark-shell']
Error: Master must start with yarn, mesos, spark, or local
Run with --help for usage help or --verbose for debug output
Traceback (most recent call last):
  File "/tmp/spark-1.0.0-rc10/python/pyspark/shell.py", line 41, in <module>
    SparkContext.setSystemProperty("spark.executor.uri", os.environ["SPARK_EXECUTOR_URI"])
  File "/tmp/spark-1.0.0-rc10/python/pyspark/context.py", line 203, in setSystemProperty
    SparkContext._ensure_initialized()
  File "/tmp/spark-1.0.0-rc10/python/pyspark/context.py", line 180, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway()
  File "/tmp/spark-1.0.0-rc10/python/pyspark/java_gateway.py", line 52, in launch_gateway
    gateway_port = int(proc.stdout.readline())
ValueError: invalid literal for int() with base 10: ''
>>>

either this assignment:

     export MASTER=mesos://10.33.9.56:5050

Or this one:

     export MASTER=mesos://zk://10.32.43.173:2181,10.33.9.56:2181,10.116.167.95:2181/mesos

Is necessary to start bin/pyspark. Is specifying a zk:// URI no longer supported intentionally, or is this a regression?



--
This message was sent by Atlassian JIRA
(v6.2#6252)