You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Tzahi File <tz...@ironsrc.com> on 2020/08/31 14:48:57 UTC

Adding Partioned Field to The File

Hi,

I'm using PySpark to write df to s3 in parquet.
I would like to add the partitioned columns to the file as well.
What is the best way to do this?

e.g df.write.partitionBy('day','hour')....
file out come -> day,hour,time,name....
and not time,name....


Thanks!
Tzahi