incremental broadcast join

["Mendelson, Assaf" <As...@rsa.com>]

Hi,

It seems as if when doing broadcast join, the entire dataframe is resent even if part of it has already been broadcasted.

Consider the following case:

val df1 = ???
val df2 = ???
val df3 = ???

df3.join(broadcast(df1), on=cond, "left_outer")
followed by
df4.join(broadcast(df1.union(df2), on=cond, "left_outer")

I would expect the second broadcast to only broadcast the difference. However, if I do explain(true) I see the entire union is broadcast.

My use case is that I have a series of dataframes on which I need to do some enrichment, joining them with a small dataframe. The small dataframe gets additional information (as the result of each aggregation).

Is there an efficient way of doing this?

Thanks,
              Assaf.