confusing about start ipython notebook with spark between 1.3.x and 1.6.x

[charles li <ch...@gmail.com>]

I used to use spark 1.3.x before, and explore my data in ipython [3.2]
notebook, which was very stable. but I came across an error

 " Java gateway process exited before sending the driver its port number "

my code is as bellow:

```
import pyspark
from pyspark import SparkConf

sc_conf = SparkConf()             ### error occurs here
```

then I ask google for help, here is a answer on stackoverflow:
http://stackoverflow.com/questions/30763951/spark-context-sc-not-defined/30851037#30851037
, it says:

```

One solution is adding pyspark-shell to the shell environment variable
PYSPARK_SUBMIT_ARGS:

export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell"

There is a change in python/pyspark/java_gateway.py
<https://github.com/apache/spark/commit/e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a#diff-d88dad45f458b7ad9f1c6a63b07e3a28>,
which requires PYSPARK_SUBMIT_ARGS includes pyspark-shell if a
PYSPARK_SUBMIT_ARGS variable is set by a user.

```

then I change my  PYSPARK_SUBMIT_ARGS from `--master spark://
10.21.208.21:7077 --deploy-mode client` to `--master spark://
10.21.208.21:7077 --deploy-mode client  pyspark-shell` , it does works, but
it raise another question, each time when I create sc in different
notebooks, the spark app name is `pyspark-shell` even though I explicitily
set the app name using SparkConf, that's really confused me these days.

then My questions come:



   - How to start ipython notebook with spark integrated in spark 1.6.0;
   - why it works when I set `pyspark-shell` in PYSPARK_SUBMIT_ARGS when
   start ipython notebook with spark 1.6.0;
   - why it does not work when I explicitly set the app name using
   SparkConf;


great thanks.

-- 
*--------------------------------------*
a spark lover, a quant, a developer and a good man.

http://github.com/litaotao