You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by Rafał Wojdyła <ra...@gmail.com> on 2023/12/10 16:13:12 UTC

Parquet MemoryManager starves Spark's DynamicPartitionDataSingleWriter?

Hi,
I would love to get your input on an issue we have stumbled upon,
originally reported in https://issues.apache.org/jira/browse/SPARK-44003.
The issue appears to be that:

> a combination of a large number of writers created via Spark's
DynamicPartitionDataSingleWriter and the global Parquet MemoryManager
synchronization bottleneck drastically reduces the performance by starving
the writer threads

Is this a known issue/"feature"? What would you suggest to workaround this
issue?
Cheers - Rafal