You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "Saha, Daniel" <da...@amazon.com.INVALID> on 2024/05/15 16:33:19 UTC

How to provide a Zstd "training mode" dictionary object

Hi,

I understand that Zstd compression can optionally be provided a dictionary object to improve performance. See “training mode” here https://facebook.github.io/zstd/

Does Spark surface a way to provide this dictionary object when writing/reading data? What about for intermediate shuffle results?

Thanks,
Daniel