You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Charles Hayden <ch...@atigeo.com> on 2015/05/13 02:40:30 UTC
how to set random seed
In pySpark, I am writing a map with a lambda that calls random.shuffle.
For testing, I want to be able to give it a seed, so that successive runs will produce the same shuffle.
I am looking for a way to set this same random seed once on each worker. Is there any simple way to do it??
Re: how to set random seed
Posted by Davies Liu <da...@databricks.com>.
The python workers used for each stage may be different, this may not
work as expected.
You can create a Random object, set the seed, use it to do the shuffle().
r = random.Random()
r.seek(my_seed)
def f(x):
r.shuffle(l)
rdd.map(f)
On Thu, May 14, 2015 at 6:21 AM, Charles Hayden
<ch...@atigeo.com> wrote:
> Thanks for the reply.
>
>
> I have not tried it out (I will today and report on my results) but I think
> what I need to do is to call mapPartitions and pass it a function that sets
> the seed. I was planning to pass the seed value in the closure.
>
>
> Something like:
>
> my_seed = 42
> def f(iterator):
> random.seed(my_seed)
> yield my_seed
> rdd.mapPartitions(f)
>
>
> ________________________________
> From: ayan guha <gu...@gmail.com>
> Sent: Thursday, May 14, 2015 2:29 AM
>
> To: Charles Hayden
> Cc: user
> Subject: Re: how to set random seed
>
> Sorry for late reply.
>
> Here is what I was thinking....
>
> import random as r
> def main():
> ....get SparkContext
> #Just for fun, lets assume seed is an id
> filename="bin.dat"
> seed = id(filename)
> #broadcast it
> br = sc.broadcast(seed)
>
> #set up dummy list
> lst = []
> for i in range(4):
> x=[]
> for j in range(4):
> x.append(j)
> lst.append(x)
> print lst
> base = sc.parallelize(lst)
> print base.map(randomize).collect()
>
> Randomize looks like
> def randomize(lst):
> local_seed = br.value
> r.seed(local_seed)
> r.shuffle(lst)
> return lst
>
>
> Let me know if this helps...
>
>
>
>
> base = sc.parallelize(lst)
> print base.map(randomize).collect()
>
> On Wed, May 13, 2015 at 11:41 PM, Charles Hayden <ch...@atigeo.com>
> wrote:
>>
>> Can you elaborate? Broadcast will distribute the seed, which is only one
>> number. But what construct do I use to "plant" the seed (call
>> random.seed()) once on each worker?
>>
>> ________________________________
>> From: ayan guha <gu...@gmail.com>
>> Sent: Tuesday, May 12, 2015 11:17 PM
>> To: Charles Hayden
>> Cc: user
>> Subject: Re: how to set random seed
>>
>>
>> Easiest way is to broadcast it.
>>
>> On 13 May 2015 10:40, "Charles Hayden" <ch...@atigeo.com> wrote:
>>>
>>> In pySpark, I am writing a map with a lambda that calls random.shuffle.
>>> For testing, I want to be able to give it a seed, so that successive runs
>>> will produce the same shuffle.
>>> I am looking for a way to set this same random seed once on each worker.
>>> Is there any simple way to do it?
>>>
>>>
>
>
>
> --
> Best Regards,
> Ayan Guha
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: how to set random seed
Posted by Charles Hayden <ch...@atigeo.com>.
Thanks for the reply.
I have not tried it out (I will today and report on my results) but I think what I need to do is to call mapPartitions and pass it a function that sets the seed. I was planning to pass the seed value in the closure.
Something like:
my_seed = 42
def f(iterator):
random.seed(my_seed)
yield my_seed
rdd.mapPartitions(f)
________________________________
From: ayan guha <gu...@gmail.com>
Sent: Thursday, May 14, 2015 2:29 AM
To: Charles Hayden
Cc: user
Subject: Re: how to set random seed
Sorry for late reply.
Here is what I was thinking....
import random as r
def main():
....get SparkContext
#Just for fun, lets assume seed is an id
filename="bin.dat"
seed = id(filename)
#broadcast it
br = sc.broadcast(seed)
#set up dummy list
lst = []
for i in range(4):
x=[]
for j in range(4):
x.append(j)
lst.append(x)
print lst
base = sc.parallelize(lst)
print base.map(randomize).collect()
Randomize looks like
def randomize(lst):
local_seed = br.value
r.seed(local_seed)
r.shuffle(lst)
return lst
Let me know if this helps...
base = sc.parallelize(lst)
print base.map(randomize).collect()
On Wed, May 13, 2015 at 11:41 PM, Charles Hayden <ch...@atigeo.com>> wrote:
?Can you elaborate? Broadcast will distribute the seed, which is only one number. But what construct do I use to "plant" the seed (call random.seed()) once on each worker?
________________________________
From: ayan guha <gu...@gmail.com>>
Sent: Tuesday, May 12, 2015 11:17 PM
To: Charles Hayden
Cc: user
Subject: Re: how to set random seed
Easiest way is to broadcast it.
On 13 May 2015 10:40, "Charles Hayden" <ch...@atigeo.com>> wrote:
In pySpark, I am writing a map with a lambda that calls random.shuffle.
For testing, I want to be able to give it a seed, so that successive runs will produce the same shuffle.
I am looking for a way to set this same random seed once on each worker. Is there any simple way to do it??
--
Best Regards,
Ayan Guha
Re: how to set random seed
Posted by ayan guha <gu...@gmail.com>.
Sorry for late reply.
Here is what I was thinking....
import random as r
def main():
....get SparkContext
#Just for fun, lets assume seed is an id
filename="bin.dat"
seed = id(filename)
#broadcast it
br = sc.broadcast(seed)
#set up dummy list
lst = []
for i in range(4):
x=[]
for j in range(4):
x.append(j)
lst.append(x)
print lst
base = sc.parallelize(lst)
print base.map(randomize).collect()
Randomize looks like
def randomize(lst):
local_seed = br.value
r.seed(local_seed)
r.shuffle(lst)
return lst
Let me know if this helps...
base = sc.parallelize(lst)
print base.map(randomize).collect()
On Wed, May 13, 2015 at 11:41 PM, Charles Hayden <ch...@atigeo.com>
wrote:
> Can you elaborate? Broadcast will distribute the seed, which is only
> one number. But what construct do I use to "plant" the seed (call
> random.seed()) once on each worker?
> ------------------------------
> *From:* ayan guha <gu...@gmail.com>
> *Sent:* Tuesday, May 12, 2015 11:17 PM
> *To:* Charles Hayden
> *Cc:* user
> *Subject:* Re: how to set random seed
>
>
> Easiest way is to broadcast it.
> On 13 May 2015 10:40, "Charles Hayden" <ch...@atigeo.com> wrote:
>
>> In pySpark, I am writing a map with a lambda that calls random.shuffle.
>> For testing, I want to be able to give it a seed, so that successive runs
>> will produce the same shuffle.
>> I am looking for a way to set this same random seed once on each
>> worker. Is there any simple way to do it?
>>
>>
>>
--
Best Regards,
Ayan Guha
Re: how to set random seed
Posted by Charles Hayden <ch...@atigeo.com>.
?Can you elaborate? Broadcast will distribute the seed, which is only one number. But what construct do I use to "plant" the seed (call random.seed()) once on each worker?
________________________________
From: ayan guha <gu...@gmail.com>
Sent: Tuesday, May 12, 2015 11:17 PM
To: Charles Hayden
Cc: user
Subject: Re: how to set random seed
Easiest way is to broadcast it.
On 13 May 2015 10:40, "Charles Hayden" <ch...@atigeo.com>> wrote:
In pySpark, I am writing a map with a lambda that calls random.shuffle.
For testing, I want to be able to give it a seed, so that successive runs will produce the same shuffle.
I am looking for a way to set this same random seed once on each worker. Is there any simple way to do it??
Re: how to set random seed
Posted by ayan guha <gu...@gmail.com>.
Easiest way is to broadcast it.
On 13 May 2015 10:40, "Charles Hayden" <ch...@atigeo.com> wrote:
> In pySpark, I am writing a map with a lambda that calls random.shuffle.
> For testing, I want to be able to give it a seed, so that successive runs
> will produce the same shuffle.
> I am looking for a way to set this same random seed once on each worker.
> Is there any simple way to do it?
>
>
>