You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Oded Poncz <od...@ubimo.com> on 2013/01/06 15:07:26 UTC
External table with partitions
Is it possible to instruct hive to get only specific files from a
partitioned external table?
For example I have the following directory structure
data/dd=2012-12-31/a1.txt
data/dd=2012-12-31/a2.txt
data/dd=2012-12-31/a3.txt
data/dd=2012-12-31/a4.txt
data/dd=2012-12-31/b1.txt
data/dd=2012-12-31/b2.txt
data/dd=2012-12-31/b2.txt
Is it possible to add 2012-12-31 as a partition and tell hive to load only
the a* files to the table?
Thanks,
Re: External table with partitions
Posted by Jagat Singh <ja...@gmail.com>.
If all files are in same partition then they satisfy condition of same
value as partion column .
You cannot do with hive but can have one intermediate table and then to
move required files using glob pattern
-----------
Sent from Mobile , short and crisp.
On 07-Jan-2013 1:07 AM, "Oded Poncz" <od...@ubimo.com> wrote:
> Is it possible to instruct hive to get only specific files from a
> partitioned external table?
> For example I have the following directory structure
>
> data/dd=2012-12-31/a1.txt
> data/dd=2012-12-31/a2.txt
> data/dd=2012-12-31/a3.txt
> data/dd=2012-12-31/a4.txt
>
> data/dd=2012-12-31/b1.txt
> data/dd=2012-12-31/b2.txt
> data/dd=2012-12-31/b2.txt
>
> Is it possible to add 2012-12-31 as a partition and tell hive to load only
> the a* files to the table?
> Thanks,
>
Re: External table with partitions
Posted by be...@yahoo.com.
Sorry, I din understand your query on first look through.
Like Jagat said, you may need to go with a temp table for this.
Do a hadoop fs -cp ../../a.* <destn dir>
Create a external table with location as 'destn dir'.
CREATE EXERNAL TABLE <tmp tble name> LIKE <src table name> LOCATION '' ;
NB: I just gave the syntax from memory. please check the syntax in hive user guide.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: bejoy_ks@yahoo.com
Date: Sun, 6 Jan 2013 14:39:45
To: <us...@hive.apache.org>
Reply-To: user@hive.apache.org
Subject: Re: External table with partitions
Hi Oded
If you have created the directories manually that would come visible to the hive table only if the partitions/ sub dirs are added to the meta data using
'ALTER TABLE ... ADD PARTITION' .
Partitions are not retrieved implicitly into hive tabe even if you have a proper sub dir structure.
Similarly if you don't need a particular partition on your table permanently you can always delete them using the alter table command.
If you are intending to use a particular partition alone in your query no need to alter the partitions. Just append a where clause to the query that has scope only on the required partitions.
Hope this helps.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Oded Poncz <od...@ubimo.com>
Date: Sun, 6 Jan 2013 16:07:26
To: <us...@hive.apache.org>
Reply-To: user@hive.apache.org
Subject: External table with partitions
Is it possible to instruct hive to get only specific files from a
partitioned external table?
For example I have the following directory structure
data/dd=2012-12-31/a1.txt
data/dd=2012-12-31/a2.txt
data/dd=2012-12-31/a3.txt
data/dd=2012-12-31/a4.txt
data/dd=2012-12-31/b1.txt
data/dd=2012-12-31/b2.txt
data/dd=2012-12-31/b2.txt
Is it possible to add 2012-12-31 as a partition and tell hive to load only
the a* files to the table?
Thanks,
Re: External table with partitions
Posted by be...@yahoo.com.
Hi Oded
If you have created the directories manually that would come visible to the hive table only if the partitions/ sub dirs are added to the meta data using
'ALTER TABLE ... ADD PARTITION' .
Partitions are not retrieved implicitly into hive tabe even if you have a proper sub dir structure.
Similarly if you don't need a particular partition on your table permanently you can always delete them using the alter table command.
If you are intending to use a particular partition alone in your query no need to alter the partitions. Just append a where clause to the query that has scope only on the required partitions.
Hope this helps.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Oded Poncz <od...@ubimo.com>
Date: Sun, 6 Jan 2013 16:07:26
To: <us...@hive.apache.org>
Reply-To: user@hive.apache.org
Subject: External table with partitions
Is it possible to instruct hive to get only specific files from a
partitioned external table?
For example I have the following directory structure
data/dd=2012-12-31/a1.txt
data/dd=2012-12-31/a2.txt
data/dd=2012-12-31/a3.txt
data/dd=2012-12-31/a4.txt
data/dd=2012-12-31/b1.txt
data/dd=2012-12-31/b2.txt
data/dd=2012-12-31/b2.txt
Is it possible to add 2012-12-31 as a partition and tell hive to load only
the a* files to the table?
Thanks,