[Shyam P <sh...@gmail.com>]

Hi All,
  I need to save a huge data frame as parquet file. As it is huge its
taking several hours. To improve performance it is known I have to send it
group wise.

But when I do partition ( columns*) /groupBy(Columns*) , driver is spilling
a lot of data and performance hits a lot again.

So how to handle this situation and save one group after another.

Attaching the sample scenario of the same.

https://stackoverflow.com/questions/54416623/how-to-group-dataframe-year-wise-and-iterate-through-groups-and-send-each-year-d

Highly appreciate your help.

Thanks,
Shyam

Re: How to group dataframe year-wise and iterate through groups and send each year to dataframe to executor?

[Shyam P <sh...@gmail.com>]

Thanks a lot Roman.

But provided link as several ways to deal the problem.
Why do we need to do operation on RDD instead dataframe/dataset ?

Do I need a custom partitioner in my case , how to invoke it in spark-sql?

Can anyone provide some sample on handling skewed data with spark-sql?

Thanks,
Shyam

>