You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by 诺铁 <no...@gmail.com> on 2015/06/26 08:41:40 UTC

how to do table partitioning efficiently?

hi,

now I'm doing something like this on a data frame to make use of table
partitioning

df.filter($"sex" === "male").write.parquet("path/to/table/sex=male")
df.filter($"sex" === "female").write.parquet("path/to/table/sex=female")

this will filter dataset multiple times, are there better way to do this?


thanks.