You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by shyla deshpande <de...@gmail.com> on 2017/03/28 21:57:53 UTC

dataframe join questions?

Following are my questions. Thank you.

1. When joining dataframes is it a good idea to repartition on the key
column that is used in the join or
the optimizer is too smart so forget it.

2. In RDD join, wherever possible we do reduceByKey before the join to
avoid a big shuffle of data. Do we need
to do anything similar with dataframe joins, or the optimizer is too
smart so forget it.