You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by 诺铁 <no...@gmail.com> on 2015/06/26 08:41:40 UTC
how to do table partitioning efficiently?
hi,
now I'm doing something like this on a data frame to make use of table
partitioning
df.filter($"sex" === "male").write.parquet("path/to/table/sex=male")
df.filter($"sex" === "female").write.parquet("path/to/table/sex=female")
this will filter dataset multiple times, are there better way to do this?
thanks.