You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by md...@orange.com on 2012/03/02 10:45:36 UTC

Where clause and bucketized table

Hi folks,

I have a table T bucketized  on user_id and I am surprised to see that all the buckets are read during the execution of the following query:

SELECT * from T where user_id = X

What should I do to make sure hive will account for the bucket structure to run this query ?

Cheers,
Michael

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you.


Re: Where clause and bucketized table

Posted by Mark Grover <mg...@oanda.com>.
Hi Michael,
To add to Bejoy's answer:

As far as I understand, the hashing function used in bucketing is not persistently stored, so as per your example (SELECT * from T where user_id = X), given X, Hive couldn't figure out what bucket for table T should it be looking in.

As Bejoy pointed out, bucketing is used in sampling but another place it is used in joins (like a Bucketed map join). If the value X is coming from another table that is bucketed on the same column (user_id) and the number of buckets in that table and table T are the same or a multiple of another, the join would happen on corresponding buckets instead of the entire tables, considerably improving the performance of the query in some cases.

Not sure if that fits your bill.

Good luck!
Mark

Mark Grover, Business Intelligence Analyst
OANDA Corporation 

www: oanda.com www: fxtrade.com 
e: mgrover@oanda.com 

"Best Trading Platform" - World Finance's Forex Awards 2009. 
"The One to Watch" - Treasury Today's Adam Smith Awards 2009.

----- Original Message -----
From: "Bejoy Ks" <be...@yahoo.com>
To: user@hive.apache.org
Sent: Friday, March 2, 2012 5:58:56 AM
Subject: Re: Where clause and bucketized table



Michael 
Unlike partitions, hive doesn't choose buckets by itself based on a where clause in HQL. You need to specify the number of buckets that the query should be executing on using TABLE SAMPLE clause . 
SELECT * from T TABLESAMPLE (m OUT OF n BUCKETS ON id) where user_id = X; 



https://cwiki.apache.org/Hive/languagemanual-sampling.html 



Buckets is mostly used for sampling and your query doesn't look like one, if you are looking for grouping of data into subsets and processing only the same intead of the whole data set then may be you should consider using Partitions. 



Regards 
Bejoy.K.S 





From: "mdefoinplatel.ext@orange.com" <md...@orange.com> 
To: "user@hive.apache.org" <us...@hive.apache.org> 
Sent: Friday, March 2, 2012 3:15 PM 
Subject: Where clause and bucketized table 





Hi folks, 

I have a table T bucketized on user_id and I am surprised to see that all the buckets are read during the execution of the following query: 

SELECT * from T where user_id = X 

What should I do to make sure hive will account for the bucket structure to run this query ? 

Cheers, 
Michael _________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you. 


Re: Where clause and bucketized table

Posted by Bejoy Ks <be...@yahoo.com>.
Michael
       Unlike partitions, hive doesn't choose buckets by itself based on a where clause in HQL. You need to specify the number of buckets that the query should be executing on using TABLE SAMPLE clause . 
SELECT * from T TABLESAMPLE (m OUT OF n BUCKETS ON id)  where user_id = X;


https://cwiki.apache.org/Hive/languagemanual-sampling.html


Buckets is mostly used for sampling and your query doesn't look like one, if you are looking for grouping of data into subsets and processing only the same intead of the whole data set then may be you should consider using Partitions. 


Regards
Bejoy.K.S


________________________________
 From: "mdefoinplatel.ext@orange.com" <md...@orange.com>
To: "user@hive.apache.org" <us...@hive.apache.org> 
Sent: Friday, March 2, 2012 3:15 PM
Subject: Where clause and bucketized table
 

Hi folks,
 
I have a table T bucketized  on user_id and I am surprised to see that all the buckets are read during the execution of the following query:
 
SELECT * from T where user_id = X
 
What should I do to make sure hive will account for the bucket structure to run this query ?
 
Cheers,
Michael
_________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you.