You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by lei liu <li...@gmail.com> on 2010/08/06 05:47:50 UTC

How to merge small files

When I run below sql:  INSERT OVERWRITE TABLE tablename1
select_statement1 FROM from_statement, there are many files which size
is zero are stored to hadoop,

How can I merge these small files?

Thanks,


LiuLei

Re: How to merge small files

Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Aug 10, 2010 at 8:13 AM, lei liu <li...@gmail.com> wrote:
> Thank you for your reply.
>
> Could you tell me why it is slower if the two paremeters are true and how
> slow it is?
>
> 2010/8/10 Namit Jain <nj...@facebook.com>
>>
>> Yes, it will try to run another map-reduce job to merge the files
>> ________________________________________
>> From: lei liu [liulei412@gmail.com]
>> Sent: Monday, August 09, 2010 8:57 AM
>> To: hive-user@hadoop.apache.org
>> Subject: Re: How to merge small files
>>
>> Could you tell me whether the query is slower if I two parameters both are
>> true?
>>
>> 2010/8/9 Namit Jain <nj...@facebook.com>>
>> That's right
>>
>> ________________________________________
>> From: lei liu [liulei412@gmail.com<ma...@gmail.com>]
>> Sent: Sunday, August 08, 2010 7:18 PM
>> To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
>> Subject: Re: How to merge small files
>>
>> Thank you for your reply.
>>
>> Your mean is I will execute below statement:
>>
>> statement.execute("set hive.merge.mapfiles=true");
>> statement.execute("set hive.merge.mapredfiles=true");
>>
>> The two parementers are both true, right?
>>
>> 2010/8/6 Namit Jain
>> <nj...@facebook.com>>>
>>  HIVEMERGEMAPFILES("hive.merge.mapfiles", true),
>>  HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false),
>>
>>
>> Set the above parameters to true before your query.
>>
>>
>>
>> ________________________________________
>> From: lei liu
>> [liulei412@gmail.com<ma...@gmail.com>>]
>> Sent: Thursday, August 05, 2010 8:47 PM
>> To:
>> hive-user@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Subject: How to merge small files
>>
>> When I run below sql:  INSERT OVERWRITE TABLE tablename1 select_statement1
>> FROM from_statement, there are many files which size is zero are stored to
>> hadoop,
>>
>> How can I merge these small files?
>>
>> Thanks,
>>
>>
>>
>> LiuLei
>>
>>
>>
>
>

How slow it is is relevant to how much data you have. We can not
answer questions like that, try it both ways and find out for
yourself.

Edward

Re: How to merge small files

Posted by lei liu <li...@gmail.com>.
Thank you for your reply.

Could you tell me why it is slower if the two paremeters are true and how
slow it is?

2010/8/10 Namit Jain <nj...@facebook.com>

> Yes, it will try to run another map-reduce job to merge the files
> ________________________________________
> From: lei liu [liulei412@gmail.com]
> Sent: Monday, August 09, 2010 8:57 AM
> To: hive-user@hadoop.apache.org
> Subject: Re: How to merge small files
>
> Could you tell me whether the query is slower if I two parameters both are
> true?
>
> 2010/8/9 Namit Jain <nj...@facebook.com>>
> That's right
>
> ________________________________________
> From: lei liu [liulei412@gmail.com<ma...@gmail.com>]
> Sent: Sunday, August 08, 2010 7:18 PM
> To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
> Subject: Re: How to merge small files
>
> Thank you for your reply.
>
> Your mean is I will execute below statement:
>
> statement.execute("set hive.merge.mapfiles=true");
> statement.execute("set hive.merge.mapredfiles=true");
>
> The two parementers are both true, right?
>
> 2010/8/6 Namit Jain <nj...@facebook.com><mailto:
> njain@facebook.com<ma...@facebook.com>>>
>   HIVEMERGEMAPFILES("hive.merge.mapfiles", true),
>  HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false),
>
>
> Set the above parameters to true before your query.
>
>
>
> ________________________________________
> From: lei liu [liulei412@gmail.com<ma...@gmail.com><mailto:
> liulei412@gmail.com<ma...@gmail.com>>]
> Sent: Thursday, August 05, 2010 8:47 PM
> To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org
> ><ma...@hadoop.apache.org>>
> Subject: How to merge small files
>
> When I run below sql:  INSERT OVERWRITE TABLE tablename1 select_statement1
> FROM from_statement, there are many files which size is zero are stored to
> hadoop,
>
> How can I merge these small files?
>
> Thanks,
>
>
>
> LiuLei
>
>
>
>

Re: How to merge small files

Posted by Edward Capriolo <ed...@gmail.com>.
Lei,

Are you still using hive 4.1 or have you upgraded, the merge options
mentioned above were probable not present until 5.0

Edward

On Mon, Aug 9, 2010 at 9:59 PM, Todd Lee <ro...@gmail.com> wrote:
> as long as the files are inside the same directory, hive will treat them as a table.
>
>
> Todd
>
> On Aug 9, 2010, at 6:07 PM, "Bakshi, Ankita" <An...@ironmountain.com> wrote:
>
>>
>> Hi,
>>
>> Sorry to hijack this thread. But I am curious if there any other in-built option to merge files in the directory before loading data into the table.
>>
>> I have a directory in the local file system which contains many small files. I want to load it to a single hive table. I am wondering what would be the best approach to this problem.
>>
>> Thanks,
>> Ankita
>>
>>
>> -----Original Message-----
>> From: Namit Jain [mailto:njain@facebook.com]
>> Sent: Monday, August 09, 2010 9:32 AM
>> To: hive-user@hadoop.apache.org
>> Subject: RE: How to merge small files
>>
>> Yes, it will try to run another map-reduce job to merge the files
>> ________________________________________
>> From: lei liu [liulei412@gmail.com]
>> Sent: Monday, August 09, 2010 8:57 AM
>> To: hive-user@hadoop.apache.org
>> Subject: Re: How to merge small files
>>
>> Could you tell me whether the query is slower if I two parameters both are true?
>>
>> 2010/8/9 Namit Jain <nj...@facebook.com>>
>> That's right
>>
>> ________________________________________
>> From: lei liu [liulei412@gmail.com<ma...@gmail.com>]
>> Sent: Sunday, August 08, 2010 7:18 PM
>> To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
>> Subject: Re: How to merge small files
>>
>> Thank you for your reply.
>>
>> Your mean is I will execute below statement:
>>
>> statement.execute("set hive.merge.mapfiles=true");
>> statement.execute("set hive.merge.mapredfiles=true");
>>
>> The two parementers are both true, right?
>>
>> 2010/8/6 Namit Jain <nj...@facebook.com>>>
>>  HIVEMERGEMAPFILES("hive.merge.mapfiles", true),
>>  HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false),
>>
>>
>> Set the above parameters to true before your query.
>>
>>
>>
>> ________________________________________
>> From: lei liu [liulei412@gmail.com<ma...@gmail.com>>]
>> Sent: Thursday, August 05, 2010 8:47 PM
>> To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Subject: How to merge small files
>>
>> When I run below sql:  INSERT OVERWRITE TABLE tablename1 select_statement1 FROM from_statement, there are many files which size is zero are stored to hadoop,
>>
>> How can I merge these small files?
>>
>> Thanks,
>>
>>
>>
>> LiuLei
>>
>>
>>
>> The information contained in this email message and its attachments is intended only for the private and confidential use of the recipient(s) named above, unless the sender expressly agrees otherwise. Transmission of email over the Internet is not a secure communications medium. If you are requesting or have requested the transmittal of personal data, as defined in applicable privacy laws by means of email or in an attachment to email, you must select a more secure alternate means of transmittal that supports your obligations to protect such personal data. If the reader of this message is not the intended recipient and/or you have received this email in error, you must take no action based on the information in this email and you are hereby notified that any dissemination, misuse or copying or disclosure of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by email and delete the original message.
>>
>

Re: How to merge small files

Posted by Todd Lee <ro...@gmail.com>.
as long as the files are inside the same directory, hive will treat them as a table.


Todd

On Aug 9, 2010, at 6:07 PM, "Bakshi, Ankita" <An...@ironmountain.com> wrote:

> 
> Hi,
> 
> Sorry to hijack this thread. But I am curious if there any other in-built option to merge files in the directory before loading data into the table.
> 
> I have a directory in the local file system which contains many small files. I want to load it to a single hive table. I am wondering what would be the best approach to this problem.
> 
> Thanks,
> Ankita
> 
> 
> -----Original Message-----
> From: Namit Jain [mailto:njain@facebook.com] 
> Sent: Monday, August 09, 2010 9:32 AM
> To: hive-user@hadoop.apache.org
> Subject: RE: How to merge small files
> 
> Yes, it will try to run another map-reduce job to merge the files
> ________________________________________
> From: lei liu [liulei412@gmail.com]
> Sent: Monday, August 09, 2010 8:57 AM
> To: hive-user@hadoop.apache.org
> Subject: Re: How to merge small files
> 
> Could you tell me whether the query is slower if I two parameters both are true?
> 
> 2010/8/9 Namit Jain <nj...@facebook.com>>
> That's right
> 
> ________________________________________
> From: lei liu [liulei412@gmail.com<ma...@gmail.com>]
> Sent: Sunday, August 08, 2010 7:18 PM
> To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
> Subject: Re: How to merge small files
> 
> Thank you for your reply.
> 
> Your mean is I will execute below statement:
> 
> statement.execute("set hive.merge.mapfiles=true");
> statement.execute("set hive.merge.mapredfiles=true");
> 
> The two parementers are both true, right?
> 
> 2010/8/6 Namit Jain <nj...@facebook.com>>>
>  HIVEMERGEMAPFILES("hive.merge.mapfiles", true),
>  HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false),
> 
> 
> Set the above parameters to true before your query.
> 
> 
> 
> ________________________________________
> From: lei liu [liulei412@gmail.com<ma...@gmail.com>>]
> Sent: Thursday, August 05, 2010 8:47 PM
> To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: How to merge small files
> 
> When I run below sql:  INSERT OVERWRITE TABLE tablename1 select_statement1 FROM from_statement, there are many files which size is zero are stored to hadoop,
> 
> How can I merge these small files?
> 
> Thanks,
> 
> 
> 
> LiuLei
> 
> 
> 
> The information contained in this email message and its attachments is intended only for the private and confidential use of the recipient(s) named above, unless the sender expressly agrees otherwise. Transmission of email over the Internet is not a secure communications medium. If you are requesting or have requested the transmittal of personal data, as defined in applicable privacy laws by means of email or in an attachment to email, you must select a more secure alternate means of transmittal that supports your obligations to protect such personal data. If the reader of this message is not the intended recipient and/or you have received this email in error, you must take no action based on the information in this email and you are hereby notified that any dissemination, misuse or copying or disclosure of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by email and delete the original message. 
>

RE: How to merge small files

Posted by "Bakshi, Ankita" <An...@ironmountain.com>.
Hi,

Sorry to hijack this thread. But I am curious if there any other in-built option to merge files in the directory before loading data into the table.

I have a directory in the local file system which contains many small files. I want to load it to a single hive table. I am wondering what would be the best approach to this problem.

Thanks,
Ankita


-----Original Message-----
From: Namit Jain [mailto:njain@facebook.com] 
Sent: Monday, August 09, 2010 9:32 AM
To: hive-user@hadoop.apache.org
Subject: RE: How to merge small files

Yes, it will try to run another map-reduce job to merge the files
________________________________________
From: lei liu [liulei412@gmail.com]
Sent: Monday, August 09, 2010 8:57 AM
To: hive-user@hadoop.apache.org
Subject: Re: How to merge small files

Could you tell me whether the query is slower if I two parameters both are true?

2010/8/9 Namit Jain <nj...@facebook.com>>
That's right

________________________________________
From: lei liu [liulei412@gmail.com<ma...@gmail.com>]
Sent: Sunday, August 08, 2010 7:18 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: How to merge small files

Thank you for your reply.

Your mean is I will execute below statement:

statement.execute("set hive.merge.mapfiles=true");
statement.execute("set hive.merge.mapredfiles=true");

The two parementers are both true, right?

2010/8/6 Namit Jain <nj...@facebook.com>>>
  HIVEMERGEMAPFILES("hive.merge.mapfiles", true),
  HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false),


Set the above parameters to true before your query.



________________________________________
From: lei liu [liulei412@gmail.com<ma...@gmail.com>>]
Sent: Thursday, August 05, 2010 8:47 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>>
Subject: How to merge small files

When I run below sql:  INSERT OVERWRITE TABLE tablename1 select_statement1 FROM from_statement, there are many files which size is zero are stored to hadoop,

How can I merge these small files?

Thanks,



LiuLei



The information contained in this email message and its attachments is intended only for the private and confidential use of the recipient(s) named above, unless the sender expressly agrees otherwise. Transmission of email over the Internet is not a secure communications medium. If you are requesting or have requested the transmittal of personal data, as defined in applicable privacy laws by means of email or in an attachment to email, you must select a more secure alternate means of transmittal that supports your obligations to protect such personal data. If the reader of this message is not the intended recipient and/or you have received this email in error, you must take no action based on the information in this email and you are hereby notified that any dissemination, misuse or copying or disclosure of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by email and delete the original message.

RE: How to merge small files

Posted by Namit Jain <nj...@facebook.com>.
Yes, it will try to run another map-reduce job to merge the files
________________________________________
From: lei liu [liulei412@gmail.com]
Sent: Monday, August 09, 2010 8:57 AM
To: hive-user@hadoop.apache.org
Subject: Re: How to merge small files

Could you tell me whether the query is slower if I two parameters both are true?

2010/8/9 Namit Jain <nj...@facebook.com>>
That's right

________________________________________
From: lei liu [liulei412@gmail.com<ma...@gmail.com>]
Sent: Sunday, August 08, 2010 7:18 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: How to merge small files

Thank you for your reply.

Your mean is I will execute below statement:

statement.execute("set hive.merge.mapfiles=true");
statement.execute("set hive.merge.mapredfiles=true");

The two parementers are both true, right?

2010/8/6 Namit Jain <nj...@facebook.com>>>
  HIVEMERGEMAPFILES("hive.merge.mapfiles", true),
  HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false),


Set the above parameters to true before your query.



________________________________________
From: lei liu [liulei412@gmail.com<ma...@gmail.com>>]
Sent: Thursday, August 05, 2010 8:47 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>>
Subject: How to merge small files

When I run below sql:  INSERT OVERWRITE TABLE tablename1 select_statement1 FROM from_statement, there are many files which size is zero are stored to hadoop,

How can I merge these small files?

Thanks,



LiuLei

Re: How to merge small files

Posted by lei liu <li...@gmail.com>.
Could you tell me whether the query is slower if I two parameters both are
true?

2010/8/9 Namit Jain <nj...@facebook.com>

> That's right
>
> ________________________________________
> From: lei liu [liulei412@gmail.com]
> Sent: Sunday, August 08, 2010 7:18 PM
> To: hive-user@hadoop.apache.org
> Subject: Re: How to merge small files
>
> Thank you for your reply.
>
> Your mean is I will execute below statement:
>
> statement.execute("set hive.merge.mapfiles=true");
> statement.execute("set hive.merge.mapredfiles=true");
>
> The two parementers are both true, right?
>
> 2010/8/6 Namit Jain <nj...@facebook.com>>
>   HIVEMERGEMAPFILES("hive.merge.mapfiles", true),
>   HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false),
>
>
> Set the above parameters to true before your query.
>
>
>
> ________________________________________
> From: lei liu [liulei412@gmail.com<ma...@gmail.com>]
> Sent: Thursday, August 05, 2010 8:47 PM
> To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
>  Subject: How to merge small files
>
> When I run below sql:  INSERT OVERWRITE TABLE tablename1 select_statement1
> FROM from_statement, there are many files which size is zero are stored to
> hadoop,
>
> How can I merge these small files?
>
> Thanks,
>
>
>
> LiuLei
>
>
>

RE: How to merge small files

Posted by Namit Jain <nj...@facebook.com>.
That's right

________________________________________
From: lei liu [liulei412@gmail.com]
Sent: Sunday, August 08, 2010 7:18 PM
To: hive-user@hadoop.apache.org
Subject: Re: How to merge small files

Thank you for your reply.

Your mean is I will execute below statement:

statement.execute("set hive.merge.mapfiles=true");
statement.execute("set hive.merge.mapredfiles=true");

The two parementers are both true, right?

2010/8/6 Namit Jain <nj...@facebook.com>>
   HIVEMERGEMAPFILES("hive.merge.mapfiles", true),
   HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false),


Set the above parameters to true before your query.



________________________________________
From: lei liu [liulei412@gmail.com<ma...@gmail.com>]
Sent: Thursday, August 05, 2010 8:47 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: How to merge small files

When I run below sql:  INSERT OVERWRITE TABLE tablename1 select_statement1 FROM from_statement, there are many files which size is zero are stored to hadoop,

How can I merge these small files?

Thanks,



LiuLei

Re: How to merge small files

Posted by lei liu <li...@gmail.com>.
Thank you for your reply.

Your mean is I will execute below statement:

statement.execute("set hive.merge.mapfiles=true");
statement.execute("set hive.merge.mapredfiles=true");

The two parementers are both true, right?

2010/8/6 Namit Jain <nj...@facebook.com>

>    HIVEMERGEMAPFILES("hive.merge.mapfiles", true),
>    HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false),
>
>
> Set the above parameters to true before your query.
>
>
>
> ________________________________________
> From: lei liu [liulei412@gmail.com]
> Sent: Thursday, August 05, 2010 8:47 PM
> To: hive-user@hadoop.apache.org
> Subject: How to merge small files
>
> When I run below sql:  INSERT OVERWRITE TABLE tablename1 select_statement1
> FROM from_statement, there are many files which size is zero are stored to
> hadoop,
>
> How can I merge these small files?
>
> Thanks,
>
>
>
> LiuLei
>
>

RE: How to merge small files

Posted by Namit Jain <nj...@facebook.com>.
    HIVEMERGEMAPFILES("hive.merge.mapfiles", true),
    HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false),


Set the above parameters to true before your query.



________________________________________
From: lei liu [liulei412@gmail.com]
Sent: Thursday, August 05, 2010 8:47 PM
To: hive-user@hadoop.apache.org
Subject: How to merge small files

When I run below sql:  INSERT OVERWRITE TABLE tablename1 select_statement1 FROM from_statement, there are many files which size is zero are stored to hadoop,

How can I merge these small files?

Thanks,



LiuLei