You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mark Grover <mg...@oanda.com> on 2011/09/15 23:16:41 UTC

Compression and Indexing

Hi all,
I've a question regarding compression and indexing.

I would like to compress our Hive data (presently present as 
SequenceFile). Also, I have an index on this table and would like to 
maintain the index as well (i.e. keep using it).

Question 1:
Sequence file compression can be block or record based. For indexing to 
work, do I need to have block based compression? If both block and 
record based compression can work with indexing, can someone provide 
insight into which to use when?

Question 2:
BZip2 is also a block based compression and is splittable in Hadoop. Do 
you see any issues with storing data in BZip2 files and using indexing 
on that data?

Question 3 (and perhaps, the most important):
What are the best practices for compression (with or without indexing). 
Are folks typically using Sequence File compression as compared to other 
compressions (like BZip2)? If using Sequence File compression, are folks 
using record based or block based?


Thank you in advance!
Mark

-- 
Mark Grover, Business Intelligence Analyst
OANDA Corporation

www: oanda.com www: fxtrade.com
e: mgrover@oanda.com

"Best Trading Platform" - World Finance's Forex Awards 2009.
"The One to Watch" - Treasury Today's Adam Smith Awards 2009.


Re: Compression and Indexing

Posted by Mark Grover <mg...@oanda.com>.
Thanks, Yongqiang!

Could you please confirm my understanding of how to use block compression?

As of now, I am setting these properties before populating the table 
that should contain compressed data:
SET io.seqfile.compression.type=BLOCK;
SET hive.exec.compress.output=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;

Question 1:
Do I need to set io.seqfile.compress.blocksize? If so, to what? It's set 
to 1000000 by default

Question 2:
Do I need to set hive.merge.mapfiles? If so, to what? It's set to true 
by default.

Question 3:
Any other options I need to set up?

Thanks again!

Mark
P.S: I am using Hive 0.7.1 with Hadoop 0.20

On 11-09-15 05:41 PM, yongqiang he wrote:
>>> Question 1:
> Indexing should work for both. But i suggest u use block compression.
>
>>> Question 3 (and perhaps, the most important):
> block based compression.
>
>
> On Thu, Sep 15, 2011 at 2:16 PM, Mark Grover<mg...@oanda.com>  wrote:
>> Hi all,
>> I've a question regarding compression and indexing.
>>
>> I would like to compress our Hive data (presently present as SequenceFile).
>> Also, I have an index on this table and would like to maintain the index as
>> well (i.e. keep using it).
>>
>> Question 1:
>> Sequence file compression can be block or record based. For indexing to
>> work, do I need to have block based compression? If both block and record
>> based compression can work with indexing, can someone provide insight into
>> which to use when?
>>
>> Question 2:
>> BZip2 is also a block based compression and is splittable in Hadoop. Do you
>> see any issues with storing data in BZip2 files and using indexing on that
>> data?
>>
>> Question 3 (and perhaps, the most important):
>> What are the best practices for compression (with or without indexing). Are
>> folks typically using Sequence File compression as compared to other
>> compressions (like BZip2)? If using Sequence File compression, are folks
>> using record based or block based?
>>
>>
>> Thank you in advance!
>> Mark
>>
>> --
>> Mark Grover, Business Intelligence Analyst
>> OANDA Corporation
>>
>> www: oanda.com www: fxtrade.com
>> e: mgrover@oanda.com
>>
>> "Best Trading Platform" - World Finance's Forex Awards 2009.
>> "The One to Watch" - Treasury Today's Adam Smith Awards 2009.
>>
>>

-- 
Mark Grover, Business Intelligence Analyst
OANDA Corporation

www: oanda.com www: fxtrade.com
e: mgrover@oanda.com

"Best Trading Platform" - World Finance's Forex Awards 2009.
"The One to Watch" - Treasury Today's Adam Smith Awards 2009.


Re: Compression and Indexing

Posted by yongqiang he <he...@gmail.com>.
>>Question 1:
Indexing should work for both. But i suggest u use block compression.

>>Question 3 (and perhaps, the most important):
block based compression.


On Thu, Sep 15, 2011 at 2:16 PM, Mark Grover <mg...@oanda.com> wrote:
> Hi all,
> I've a question regarding compression and indexing.
>
> I would like to compress our Hive data (presently present as SequenceFile).
> Also, I have an index on this table and would like to maintain the index as
> well (i.e. keep using it).
>
> Question 1:
> Sequence file compression can be block or record based. For indexing to
> work, do I need to have block based compression? If both block and record
> based compression can work with indexing, can someone provide insight into
> which to use when?
>
> Question 2:
> BZip2 is also a block based compression and is splittable in Hadoop. Do you
> see any issues with storing data in BZip2 files and using indexing on that
> data?
>
> Question 3 (and perhaps, the most important):
> What are the best practices for compression (with or without indexing). Are
> folks typically using Sequence File compression as compared to other
> compressions (like BZip2)? If using Sequence File compression, are folks
> using record based or block based?
>
>
> Thank you in advance!
> Mark
>
> --
> Mark Grover, Business Intelligence Analyst
> OANDA Corporation
>
> www: oanda.com www: fxtrade.com
> e: mgrover@oanda.com
>
> "Best Trading Platform" - World Finance's Forex Awards 2009.
> "The One to Watch" - Treasury Today's Adam Smith Awards 2009.
>
>