Hive job tuning

[Ranjith <ra...@gmail.com>]

I have been looking at a job that was performing badly. Noticed there were several splits occurring due to the buffer record limit being reached. I get that the io.sort.mb provides the data and record buffer for the mapper task. Given that the mapper jvm starts up with 500mb and the buffer is 300, where does the rest of the 200 mb get assigned to? Is sitting there idle? If I increase the io.sort mb to the size of the jvm what effects does that have?

Thanks,
Ranjith