the number of files after merging

[patcharee <Pa...@uni.no>]

Hi,

I am using alter command below to merge partitioned orc file on one 
partition:

alter table X partition(zone=1,z=1,year=2009,month=1) CONCATENATE;

- How can I control the number of files after merging? I would like to 
get only one file per partition.
- Is it possible to concatenate the whole table, not one-by-one partition?

Thanks,
Patcharee

Re: the number of files after merging

[Prasanth Jayachandran <pj...@hortonworks.com>]

Hi

CONCATENATE uses CombineHiveInputFormat internally to group files for concatenate. Many small files are grouped until the total size reaches default max split size of 256MB. These files are then merged to form a single file. So if you want to control the number of files you have to bump up the values of mapred.max.split.size or mapreduce.input.fileinputformat.split.maxsize to higher values until you get desired number of files.

> - Is it possible to concatenate the whole table, not one-by-one partition?

Unfortunately, at this point no. I already have this feature in my to-do list. I will see if I can get this into next release.

Thanks
Prasanth

> On Oct 22, 2015, at 1:43 AM, patcharee <Pa...@uni.no> wrote:
> 
> Hi,
> 
> I am using alter command below to merge partitioned orc file on one 
> partition:
> 
> alter table X partition(zone=1,z=1,year=2009,month=1) CONCATENATE;
> 
> - How can I control the number of files after merging? I would like to 
> get only one file per partition.
> - Is it possible to concatenate the whole table, not one-by-one partition?
> 
> Thanks,
> Patcharee
> 
>