You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by jo...@accenture.com on 2011/07/08 03:06:44 UTC

Lzo compression on Hive table

Hi there,
I got my hadoop all setup writing out sequence file with LZO compression.  Using the following:
mapred.output.compress=true
mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec

How do I define my table so it will write out compressed data and be able to read in compressed data during my Hive queries?

CREATE EXTERNAL TABLE foo (
                       columnA string,
                       columnB string )
       PARTITIONED BY (date string)
       ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
       LOCATION '/path/to/hive/tables/foo';

Thanks,
Jon

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information.  If you have received it in error, please notify the sender immediately and delete the original.  Any other use of the email by you is prohibited.

Re: Lzo compression on Hive table

Posted by Aaron Sun <aa...@gmail.com>.
You can use this one

STORED AS INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"


On Thu, Jul 7, 2011 at 6:06 PM, <jo...@accenture.com> wrote:

> Hi there,
> I got my hadoop all setup writing out sequence file with LZO compression.
>  Using the following:
> mapred.output.compress=true
> mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec
>
> How do I define my table so it will write out compressed data and be able
> to read in compressed data during my Hive queries?
>
> CREATE EXTERNAL TABLE foo (
>                       columnA string,
>                       columnB string )
>       PARTITIONED BY (date string)
>       ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
>       LOCATION '/path/to/hive/tables/foo';
>
> Thanks,
> Jon
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information.  If you have
> received it in error, please notify the sender immediately and delete the
> original.  Any other use of the email by you is prohibited.
>

RE: Lzo compression on Hive table

Posted by Steven Wong <sw...@netflix.com>.
When writing, set hive.exec.compress.output=true also.

When reading, nothing special needs to be done.


-----Original Message-----
From: jonathan.hwang@accenture.com [mailto:jonathan.hwang@accenture.com] 
Sent: Thursday, July 07, 2011 6:07 PM
To: user@hive.apache.org
Subject: Lzo compression on Hive table

Hi there,
I got my hadoop all setup writing out sequence file with LZO compression.  Using the following:
mapred.output.compress=true
mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec

How do I define my table so it will write out compressed data and be able to read in compressed data during my Hive queries?

CREATE EXTERNAL TABLE foo (
                       columnA string,
                       columnB string )
       PARTITIONED BY (date string)
       ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
       LOCATION '/path/to/hive/tables/foo';

Thanks,
Jon

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information.  If you have received it in error, please notify the sender immediately and delete the original.  Any other use of the email by you is prohibited.