How to compress the text file - LZO utility ?

[Raj Hadoop <ha...@yahoo.com>]

Hi,

I have a large set of text files. I have created a Hive table pointing to each of these text files. I am looking to compress the files to save storage.

1) How should I compress the file to use LZO compression.

2) How to know whether LZO compression utility (command ?) is installed on the Hadoop cluster?

3) Should the Hive table definition be modified as a Sequence File if I compress the text file?

Please advise.

Thanks,
Raj

Re: How to compress the text file - LZO utility ?

[Nitin Pawar <ni...@gmail.com>]

1) How should I compress the file to use LZO compression.
a) Write your own mapreduce code
b) use pig scripts
c) create temp tables and load data in compression backed table

2) How to know whether LZO compression utility (command ?) is installed on
the Hadoop cluster?
check hadoop conf files and check which compression formats have been
enabled

3) Should the Hive table definition be modified as a Sequence File if I
compress the text file?
I did not understand this question.
SequenceFileFormat is a different file format together. Just because you
compress a text file with LZO compression, will it make to
SequenceFileFormat, personally I don't think so as I never heard
compression format changing file format.


On Tue, Dec 10, 2013 at 2:40 AM, Raj Hadoop <ha...@yahoo.com> wrote:

> Hi,
>
> I have a large set of text files. I have created a Hive table pointing to
> each of these text files. I am looking to compress the files to save
> storage.
>
> 1) How should I compress the file to use LZO compression.
>
> 2) How to know whether LZO compression utility (command ?) is installed on
> the Hadoop cluster?
>
> 3) Should the Hive table definition be modified as a Sequence File if I
> compress the text file?
>
> Please advise.
>
> Thanks,
> Raj
>
>


-- 
Nitin Pawar