You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Shrikanth J R <sh...@thinkdeeply.com> on 2021/12/02 15:35:26 UTC

Exploding huge array elements in spark

Hi,

I am facing an issue when exploding multiple array columns where few
columns have a huge element list. Repartition dataset before explode
doesn't work Getting memory and space issue though file size before explode
is ~16mb and Ram is 32gb with ~150gb. What is the efficient way of handling

Regards,
Shrikanth R.

Re: Exploding huge array elements in spark

Posted by Gourav Sengupta <go...@gmail.com>.

Hi Srikanth,

what is the spark version that you are using?

Can you tell us the data dictionary and the PK? Also if possible the data
volumes that you are dealing with?

Thanks and Regards,
Gourav Sengupta

On Thu, Dec 2, 2021 at 4:33 PM Shrikanth J R <sh...@thinkdeeply.com>
wrote:

> Hi,
>
> I am facing an issue when exploding multiple array columns where few
> columns have a huge element list. Repartition dataset before explode
> doesn't work Getting memory and space issue though file size before explode
> is ~16mb and Ram is 32gb with ~150gb. What is the efficient way of handling
>
> Regards,
> Shrikanth R.
>