You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Perfect Stranger <pa...@gmail.com> on 2024/09/25 16:36:18 UTC

When exactly does AQE coalesce small partitions or break up large partitions?

When exactly does AQE coalesce small partitions or break up large
partitions? Say, I do df.repartition(2), and the data in this df is huge,
will spark first repartition into 2 partitions (which may cause an OOM) and
then break them up? Or will it make the necessary number of partitions
according to the advisoryPartitionSizeInBytes setting right away, without
repartitioning into 2 partitions first?

Thanks!