You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by David Ginzburg <gi...@hotmail.com> on 2011/06/29 17:33:44 UTC
Small file problem and GenMRFileSink1
Hi,
I'm not sure weather this belongs in the hive-dev or hive-user.
I have a folder with many small files.
I would like to reduce the number of files the way hive merges output .
I tried to understand from the source of org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1 how to leverage the API to submit a job
that merges output files.
I think I was able to identify:
private void createMergeJob(FileSinkOperator fsOp, GenMRProcContext ctx, String finalName)
throws SemanticException
As the entry point to the logic that performs the operation, but I did not find documentation as to how to use it
Is there an example that simulates the use of this API call?
Re: Small file problem and GenMRFileSink1
Posted by Ning Zhang <nz...@fb.com>.
If you are using hive trunk and your table is stored in RCFile format, you can run
alter table src_rc_merge_test concatenate;
On Jun 30, 2011, at 9:53 AM, David Ginzburg wrote:
>
>
> Hi,
> I'm not sure weather this belongs in the hive-dev or hive-user.
> I have a folder with many small files.
> I would like to reduce the number of files the way hive merges output .
> I tried to understand from the source of org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1 how to leverage the API to submit a job
> that merges output files.
> I think I was able to identify:
> private void createMergeJob(FileSinkOperator fsOp, GenMRProcContext ctx, String finalName)
> throws SemanticException
> As the entry point to the logic that performs the operation, but I did not find documentation as to how to use it
>
> Is there an example that simulates the use of this API call?
>
>
>
>
>
FW: Small file problem and GenMRFileSink1
Posted by David Ginzburg <gi...@hotmail.com>.
Hi,
I'm not sure weather this belongs in the hive-dev or hive-user.
I have a folder with many small files.
I would like to reduce the number of files the way hive merges output .
I tried to understand from the source of org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1 how to leverage the API to submit a job
that merges output files.
I think I was able to identify:
private void createMergeJob(FileSinkOperator fsOp, GenMRProcContext ctx, String finalName)
throws SemanticException
As the entry point to the logic that performs the operation, but I did not find documentation as to how to use it
Is there an example that simulates the use of this API call?
RE: Small file problem and GenMRFileSink1
Posted by David Ginzburg <gi...@hotmail.com>.
What if it isn't a hive table? just an hdfs folder? can I create a temporary folder and then merge or somehow use the API that invokes the merge job ?
> From: ginzman@hotmail.com
> To: dev@hive.apache.org
> Subject: Small file problem and GenMRFileSink1
> Date: Wed, 29 Jun 2011 15:33:44 +0000
>
>
>
>
>
> Hi,
> I'm not sure weather this belongs in the hive-dev or hive-user.
> I have a folder with many small files.
> I would like to reduce the number of files the way hive merges output .
> I tried to understand from the source of org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1 how to leverage the API to submit a job
> that merges output files.
> I think I was able to identify:
> private void createMergeJob(FileSinkOperator fsOp, GenMRProcContext ctx, String finalName)
> throws SemanticException
> As the entry point to the logic that performs the operation, but I did not find documentation as to how to use it
>
> Is there an example that simulates the use of this API call?
>
>
>
>