CombineHiveInputFormat not splitting as expected

[Rob Verkuylen <ro...@verkuylen.net>]

Hi all,

We are compacting snappy compressed sequence files in our input stream on
which we run Hive queries. Generally working towards files of 50+ GB.

We see some unexpected behaviour where even smaller files, for example
5.2GB are only getting a single mapper in a Hive query using the default
CombineHiveInputFormat. Switching to HiveInputFormat gives us the 21
mappers in the Hive query which is what we would expect using 256MB block
sizes.

What would be the drawbacks for switching over to HiveInputFormat over
Combined? I would imaging more potential splits when we would have many
smaller files on the same node, which in our case would not happen that
often and we have enough resources to handle the potential extra mappers.
Is this thinking correct? Any other drawbacks?

Best regards
Rob