You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by Steven Wong <sw...@netflix.com> on 2010/08/27 02:15:11 UTC

hive.exec.compress.intermediate does not work?

I have a query that runs as 2 MR jobs:

set hive.exec.compress.intermediate = true;
set hive.exec.compress.output = true;
set mapred.output.compression.codec = org.apache.hadoop.io.compress.GzipCodec;
set mapred.output.compression.type = BLOCK;
from (
    select c
    from t1
    blah blah
) x
insert overwrite table t2
partition (part = 0)
select *
blah blah;

The output of the 1st MR job is not compressed, even though I've set hive.exec.compress.intermediate = true.

Am I missing something? Using Hive 0.5.

Thanks.
Steven

RE: hive.exec.compress.intermediate does not work?

Posted by Steven Wong <sw...@netflix.com>.

Never mind. I cannot repro it.

From: Steven Wong [mailto:swong@netflix.com]
Sent: Thursday, August 26, 2010 5:15 PM
To: hive-user@hadoop.apache.org
Subject: hive.exec.compress.intermediate does not work?

I have a query that runs as 2 MR jobs:

set hive.exec.compress.intermediate = true;
set hive.exec.compress.output = true;
set mapred.output.compression.codec = org.apache.hadoop.io.compress.GzipCodec;
set mapred.output.compression.type = BLOCK;
from (
    select c
    from t1
    blah blah
) x
insert overwrite table t2
partition (part = 0)
select *
blah blah;

The output of the 1st MR job is not compressed, even though I've set hive.exec.compress.intermediate = true.

Am I missing something? Using Hive 0.5.

Thanks.
Steven