You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Daniel, Ronald (ELS-SDG)" <R....@elsevier.com> on 2015/02/27 02:04:03 UTC

partions, SQL tables, and Parquet I/O

Short story: I want to write some parquet files so they are pre-partitioned by the same key. Then, when I read them
back in, joining the two tables on that key should be about as fast as things can be done.
Can I do that, and if so, how? I don't see how to control the partitioning of a SQL table, as opposed to PairRDDs.

Thanks,
Ron