pyspark issue

[Naveen Madhire <vm...@umail.iu.edu>]

Hi,

I am running pyspark in windows and I am seeing an error while adding
pyfiles to the sparkcontext. below is the example,

sc = SparkContext("local","Sample",pyFiles="C:/sample/yattag.zip")

this fails with no file found error for "C"


The below logic is treating the path as individual files like "C", ":" "/" etc.

https://github.com/apache/spark/blob/master/python/pyspark/context.py#l195


It works if I use Spark Conf,

sparkConf.set("spark.submit.pyFiles",*"**C:/sample/yattag.zip**"*)
sc = SparkContext("local","Sample",conf=sparkConf)


Is this an existing issue or I am not including the files in correct
way in Spark Context?


Thanks.





when I run this, I am getting

Re: pyspark issue

[Sven Krasser <kr...@gmail.com>]

It expects an iterable, and if you iterate over a string, you get the
individual characters. Use a list instead:
pyfiles=['/path/to/file']

On Mon, Jul 27, 2015 at 2:40 PM, Naveen Madhire <vm...@umail.iu.edu>
wrote:

> Hi,
>
> I am running pyspark in windows and I am seeing an error while adding
> pyfiles to the sparkcontext. below is the example,
>
> sc = SparkContext("local","Sample",pyFiles="C:/sample/yattag.zip")
>
> this fails with no file found error for "C"
>
>
> The below logic is treating the path as individual files like "C", ":" "/" etc.
>
> https://github.com/apache/spark/blob/master/python/pyspark/context.py#l195
>
>
> It works if I use Spark Conf,
>
> sparkConf.set("spark.submit.pyFiles",*"**C:/sample/yattag.zip**"*)
> sc = SparkContext("local","Sample",conf=sparkConf)
>
>
> Is this an existing issue or I am not including the files in correct way in Spark Context?
>
>
> Thanks.
>
>
>
>
>
> when I run this, I am getting
>
>


-- 
www.skrasser.com <http://www.skrasser.com/?utm_source=sig>