You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by kanchan tewary <ka...@gmail.com> on 2019/04/24 06:31:30 UTC

Handle empty partitions in pyspark

Hi All,

I have a situation where the rdd is having some empty partitions, which I
would like to identify and handle while applying mapPartitions or similar
functions. Is there a way to do this in pyspark? The method isEmpty works
on the rdd only and can not be applied.

Much appreciated.

Code block:

`list1 = [1,2,3,3,6,7,8,12,6,23,45,76,9,10]

r1 = sc.parallelize(list1,20)

def adder(iterator):
    if iterator.isEmpty():
        yield 'None'
    else:
        yield sum(iterator)

print(r1.mapPartitions(adder).collect())`

Thanks & Best Regards,
Kanchan
Data Engineer, IBM