Re: {EXT} Re: Spark sql slowness in Spark 3.0.1

[Anil Dasari <ad...@guidewire.com>]

Hello,

DF is checkpointed here. So it is written to HDFS. DF is written in paraquet format and used default parallelism.

Thanks.

From: wilson <in...@bigcount.xyz>
Date: Thursday, April 14, 2022 at 2:54 PM
To: user@spark.apache.org <us...@spark.apache.org>
Subject: {EXT} Re: Spark sql slowness in Spark 3.0.1
just curious, where to  write?


Anil Dasari wrote:
> We are upgrading spark from 2.4.7 to 3.0.1. we use spark sql (hive) to
> checkpoint data frames (intermediate data). DF write is very slow in
> 3.0.1 compared to 2.4.7.
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org