Debug Spark Streaming in PyCharm

[blbradley <br...@gmail.com>]

Hello,

I'm trying to debug a PySpark app with Kafka Streaming in PyCharm. However,
PySpark cannot find the jar dependencies for Kafka Streaming without editing
the program. I can temporarily use SparkConf to set 'spark.jars', but I'm
using Mesos for production and don't want to edit my program everytime I
want to debug. I'd like to find a way to debug without editing the source.

Here's what my PyCharm debug execution command looks like:

home/brandon/.pyenv/versions/coinspark/bin/python2.7
/opt/pycharm-community/helpers/pydev/pydevd.py --multiproc --client
127.0.0.1 --port 59042 --file /home/brandon/src/coins/coinspark/streaming.py

I might be able to use spark-submit has the command PyCharm runs, but I'm
not sure if that will work with the debugger.

Thoughts?

Cheers!
Brandon Bradley



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Debug-Spark-Streaming-in-PyCharm-tp23766.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Debug Spark Streaming in PyCharm

[Tathagata Das <td...@databricks.com>]

spark-submit does a lot of magic configurations (classpaths etc) underneath
the covers to enable pyspark to find Spark JARs, etc. I am not sure how you
can start running things directly from the PyCharm IDE. Others in the
community may be able to answer. For now the main way to run pyspark stuff
is through spark-submit, or pyspark (which uses spark-submit underneath).

On Fri, Jul 10, 2015 at 6:28 AM, blbradley <br...@gmail.com> wrote:

> Hello,
>
> I'm trying to debug a PySpark app with Kafka Streaming in PyCharm. However,
> PySpark cannot find the jar dependencies for Kafka Streaming without
> editing
> the program. I can temporarily use SparkConf to set 'spark.jars', but I'm
> using Mesos for production and don't want to edit my program everytime I
> want to debug. I'd like to find a way to debug without editing the source.
>
> Here's what my PyCharm debug execution command looks like:
>
> home/brandon/.pyenv/versions/coinspark/bin/python2.7
> /opt/pycharm-community/helpers/pydev/pydevd.py --multiproc --client
> 127.0.0.1 --port 59042 --file
> /home/brandon/src/coins/coinspark/streaming.py
>
> I might be able to use spark-submit has the command PyCharm runs, but I'm
> not sure if that will work with the debugger.
>
> Thoughts?
>
> Cheers!
> Brandon Bradley
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Debug-Spark-Streaming-in-PyCharm-tp23766.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>