Location preferences in pyspark?

[Philip Weaver <ph...@gmail.com>]

I believe what I want is the exact functionality provided by
SparkContext.makeRDD in Scala. For each element in the RDD, I want specify
a list of preferred hosts for processing that element.

It looks like this method only exists in Scala, and as far as I can tell
there is no similar functionality available in python. Is this true?

- Philip

Re: Location preferences in pyspark?

[Philip Weaver <ph...@gmail.com>]

Thanks, we decided to try to add the support ourselves :).

On Tue, Oct 20, 2015 at 6:40 PM, Jeff Zhang <zj...@gmail.com> wrote:

> Yes, I don't think there is.  You can use SparkContext.parallelize() to
> make a RDD from a list. But no location preferences supported yet.
>
> On Sat, Oct 17, 2015 at 8:42 AM, Philip Weaver <ph...@gmail.com>
> wrote:
>
>> I believe what I want is the exact functionality provided by
>> SparkContext.makeRDD in Scala. For each element in the RDD, I want specify
>> a list of preferred hosts for processing that element.
>>
>> It looks like this method only exists in Scala, and as far as I can tell
>> there is no similar functionality available in python. Is this true?
>>
>> - Philip
>>
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Location preferences in pyspark?

[Jeff Zhang <zj...@gmail.com>]

Yes, I don't think there is.  You can use SparkContext.parallelize() to
make a RDD from a list. But no location preferences supported yet.

On Sat, Oct 17, 2015 at 8:42 AM, Philip Weaver <ph...@gmail.com>
wrote:

> I believe what I want is the exact functionality provided by
> SparkContext.makeRDD in Scala. For each element in the RDD, I want specify
> a list of preferred hosts for processing that element.
>
> It looks like this method only exists in Scala, and as far as I can tell
> there is no similar functionality available in python. Is this true?
>
> - Philip
>
>


-- 
Best Regards

Jeff Zhang