Map-Joining the same small table multiple times

[Mark Schramm <ms...@tetrascend.com>]

All,

We have a small table that we use the map-join technique to join to several large tables in separate hive query scripts.

As I understand it, the map-join will do some preparatory work to get the small table into the distributed cache for the map-join.  These steps are (from my understanding):


1)      Read the small table into a hash table

2)      Serialize to a file on disk

3)      Tar the file

4)      Put the tar in the distributed cache.

Now assume that we have small table A and we map-join it to B and later map-join it to C.

I presume that hive will repeat the above preparatory steps for each map-join.  After all, it can't know that table A has not changed between scripts.

Questions:


a)      Does this work get repeated with each map-join?

b)      Any way to tell hive to re-use previous map-join small tables already cached?

c)      Any way to do something like this manually and put the file into the distributed cache manually and get the map-join to re-use it?

d)      How does hive manage such distributed cache objects?  Are they cleaned out at the end of the hive query?

Thanks,
Mark