You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by reveen joe <im...@gmail.com> on 2015/11/11 06:26:29 UTC
Small files under SequenceFile table partition directories
Hi,
Most of our Hive tables are SequenceFile tables and there are currently
many small file ranging from *1-4 MB* under the Partition directories
(created by insert-overwrite). I am assuming this is due to 2 reasons
1. Some of our tables are Bucketed and so individual files are created for
each bucket of data for a given partition.
2. The places where we have set number of Reducers, produce 1 file per
Reducer.
So, the dir structure looks like below.
/PATH/TO/TABLE/DIR/partition_column=2015-11-01//000000_0
/PATH/TO/TABLE/DIR/partition_column=2015-11-01//000001_0
/PATH/TO/TABLE/DIR/partition_column=2015-11-01//000002_0
/PATH/TO/TABLE/DIR/partition_column=2015-11-01//000003_0
............................................................................................
............................................................................................
............................................................................................
/PATH/TO/TABLE/DIR/partition_column=2015-11-01//000379_0
The Block size of the cluster is 128 MB. I know that sequence file can
store FileName as Key and FileContent as Value in sequence Files but in
this case they are independent files.
Am I right that - this would add overhead to the further processing of this
data as each file would need to spin up a JVM to start a Map Task against
that file and also because of the Disk IO overhead?
If so, what could be the best remedy to combine the small files under a
partition directory? Thank you.
Re: Small files under SequenceFile table partition directories
Posted by Chetna C <ch...@gmail.com>.
You can get rid of processing overhead by using splittable compression for
your hive table data, something like 4mc <https://github.com/carlomedas/4mc>.
Or you can use hadoop's getmerge utility to merge small files periodically.
Thanks,
Chetna Chaudhari
On 11 November 2015 at 10:56, reveen joe <im...@gmail.com> wrote:
> Hi,
>
> Most of our Hive tables are SequenceFile tables and there are currently
> many small file ranging from *1-4 MB* under the Partition directories
> (created by insert-overwrite). I am assuming this is due to 2 reasons
>
> 1. Some of our tables are Bucketed and so individual files are created for
> each bucket of data for a given partition.
>
> 2. The places where we have set number of Reducers, produce 1 file per
> Reducer.
>
> So, the dir structure looks like below.
>
> /PATH/TO/TABLE/DIR/partition_column=2015-11-01//000000_0
> /PATH/TO/TABLE/DIR/partition_column=2015-11-01//000001_0
> /PATH/TO/TABLE/DIR/partition_column=2015-11-01//000002_0
> /PATH/TO/TABLE/DIR/partition_column=2015-11-01//000003_0
>
> ............................................................................................
>
> ............................................................................................
>
> ............................................................................................
> /PATH/TO/TABLE/DIR/partition_column=2015-11-01//000379_0
>
> The Block size of the cluster is 128 MB. I know that sequence file can
> store FileName as Key and FileContent as Value in sequence Files but in
> this case they are independent files.
>
> Am I right that - this would add overhead to the further processing of
> this data as each file would need to spin up a JVM to start a Map Task
> against that file and also because of the Disk IO overhead?
>
> If so, what could be the best remedy to combine the small files under a
> partition directory? Thank you.
>
>