Any quick method to sample rdd based on one filed?

[Gavin Yue <yu...@gmail.com>]

Hey,


I have a RDD[(String,Boolean)]. I want to keep all Boolean: True rows and
randomly keep some Boolean:false rows.  And hope in the final result, the
negative ones could be 10 times more than positive ones.


What would be most efficient way to do this?

Thanks,

Re: Any quick method to sample rdd based on one filed?

[Alexey Grishchenko <pr...@gmail.com>]

In my opinion aggragate+flatMap would work faster as it would make less
passes through the data. Would work like this:

import random

def agg(x,y):
    x[0] += 1 if not y[1] else 0
    x[1] += 1 if y[1] else 0
    return x

# Source data
rdd  = sc.parallelize(xrange(100000), 5)
rdd2 = rdd.map(lambda x: (x, random.choice([True, False]))).cache()

# Calculate counts for True and False
counts = rdd2.aggregate([0,0], agg, lambda x,y: (x[0]+y[0],x[1]+y[1]))
# If filtering is needed
if counts[0]*10 > counts[1]:
    # Calculate sampling ratio
    prob0 = float(counts[1])/10.0 / float(counts[0])
    # Filter falses
    rdd2  = rdd2.flatMap(lambda x: [x] if (x[1] or x[0] and random.random()
< prob0) else [])

# Count True and False again for validation - falses should be 10% of trues
rdd2.aggregate([0,0], agg, lambda x,y: (x[0]+y[0],x[1]+y[1]))

On Fri, Aug 28, 2015 at 6:39 PM, Sonal Goyal <so...@gmail.com> wrote:

> Filter into true rdd and false rdd. Union true rdd and sample of false rdd.
> On Aug 28, 2015 2:57 AM, "Gavin Yue" <yu...@gmail.com> wrote:
>
>> Hey,
>>
>>
>> I have a RDD[(String,Boolean)]. I want to keep all Boolean: True rows and
>> randomly keep some Boolean:false rows.  And hope in the final result, the
>> negative ones could be 10 times more than positive ones.
>>
>>
>> What would be most efficient way to do this?
>>
>> Thanks,
>>
>>
>>
>>


-- 
Best regards, Alexey Grishchenko

phone: +353 (87) 262-2154
email: ProgrammerAG@gmail.com
web:   http://0x0fff.com

Re: Any quick method to sample rdd based on one filed?

[Sonal Goyal <so...@gmail.com>]

Filter into true rdd and false rdd. Union true rdd and sample of false rdd.
On Aug 28, 2015 2:57 AM, "Gavin Yue" <yu...@gmail.com> wrote:

> Hey,
>
>
> I have a RDD[(String,Boolean)]. I want to keep all Boolean: True rows and
> randomly keep some Boolean:false rows.  And hope in the final result, the
> negative ones could be 10 times more than positive ones.
>
>
> What would be most efficient way to do this?
>
> Thanks,
>
>
>
>