You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jasper Knulst <ja...@vlc.nl> on 2011/05/19 11:25:51 UTC
Can Hive 0.7 Rebuild partitions ?
Hi,
I have a partitioned external table on Hive 0.7. New subfolders are
regularly added to the base table HDFS folder.
I now have to perform this scan myself and let an external tool create new
partitions by generating and firing ALTER TABLE ADD PARTITION commands.
Is there an easier way to have hive scan the base table folder to see if
there are any new partitions around? Something like REBUILD PARTITIONS
perhaps??
Couldn't find anything similar on the Hive/LanguageManual/DDL
--
Kind Regards
Jasper
Re: Can Hive 0.7 Rebuild partitions ?
Posted by Ashish Thusoo <at...@fb.com>.
afaik there is nothing like that currently. File a feature for this on the JIRA?
Ashish
On May 19, 2011, at 2:25 AM, Jasper Knulst wrote:
> Hi,
>
> I have a partitioned external table on Hive 0.7. New subfolders are regularly added to the base table HDFS folder.
> I now have to perform this scan myself and let an external tool create new partitions by generating and firing ALTER TABLE ADD PARTITION commands.
>
> Is there an easier way to have hive scan the base table folder to see if there are any new partitions around? Something like REBUILD PARTITIONS perhaps??
>
>
>
>
> Couldn't find anything similar on the Hive/LanguageManual/DDL
>
> --
> Kind Regards
>
>
>
> Jasper
Re: Can Hive 0.7 Rebuild partitions ?
Posted by Ashutosh Chauhan <ha...@apache.org>.
Indeed a useful feature: created jira for it:
https://issues.apache.org/jira/browse/HIVE-2173
Ashutosh
On Thu, May 19, 2011 at 15:52, Roberto Congiu <ro...@openx.org> wrote:
> I agree it's useful, especially for external tables, that may be loaded by
> an external process that may 'forget' to issue a ADD PARTITION.
> A 'sync partitions' feature to sync metadata with directories would be
> really handy.
>
> On Thu, May 19, 2011 at 3:23 PM, Igor Tatarinov <ig...@decide.com> wrote:
>>
>> That's Amazon's extension to Hive and it's really handy.
>>
>> On Thu, May 19, 2011 at 2:01 PM, Tim Spence <yo...@gmail.com>
>> wrote:
>>>
>>> Is this functionality handled by ALTER TABLE [name] RECOVER PARTITIONS?
>>> Take a look at this presentation for context:
>>> http://www.slideshare.net/AmazonWebServices/aws-office-hours-amazon-elastic-mapreduce
>>>
>>> Best of luck,
>>> Tim
>>>
>>>
>>>
>>>
>>> On Thu, May 19, 2011 at 2:25 AM, Jasper Knulst <ja...@vlc.nl>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have a partitioned external table on Hive 0.7. New subfolders are
>>>> regularly added to the base table HDFS folder.
>>>> I now have to perform this scan myself and let an external tool create
>>>> new partitions by generating and firing ALTER TABLE ADD PARTITION commands.
>>>>
>>>> Is there an easier way to have hive scan the base table folder to see if
>>>> there are any new partitions around? Something like REBUILD PARTITIONS
>>>> perhaps??
>>>>
>>>>
>>>>
>>>>
>>>> Couldn't find anything similar on the Hive/LanguageManual/DDL
>>>>
>>>> --
>>>> Kind Regards
>>>>
>>>>
>>>>
>>>> Jasper
>>>
>>
>
>
>
> --
> Roberto Congiu -Data Engineer - OpenX
> 20 E Del Mar blvd, Pasadena, CA
>
>
Re: Can Hive 0.7 Rebuild partitions ?
Posted by Roberto Congiu <ro...@openx.org>.
I agree it's useful, especially for external tables, that may be loaded by
an external process that may 'forget' to issue a ADD PARTITION.
A 'sync partitions' feature to sync metadata with directories would be
really handy.
On Thu, May 19, 2011 at 3:23 PM, Igor Tatarinov <ig...@decide.com> wrote:
> That's Amazon's extension to Hive and it's really handy.
>
>
> On Thu, May 19, 2011 at 2:01 PM, Tim Spence <yo...@gmail.com>wrote:
>
>> Is this functionality handled by ALTER TABLE [name] RECOVER PARTITIONS?
>> Take a look at this presentation for context:
>> http://www.slideshare.net/AmazonWebServices/aws-office-hours-amazon-elastic-mapreduce
>>
>> Best of luck,
>> Tim
>>
>>
>>
>>
>>
>> On Thu, May 19, 2011 at 2:25 AM, Jasper Knulst <ja...@vlc.nl>wrote:
>>
>>> Hi,
>>>
>>> I have a partitioned external table on Hive 0.7. New subfolders are
>>> regularly added to the base table HDFS folder.
>>> I now have to perform this scan myself and let an external tool create
>>> new partitions by generating and firing ALTER TABLE ADD PARTITION commands.
>>>
>>> Is there an easier way to have hive scan the base table folder to see if
>>> there are any new partitions around? Something like REBUILD PARTITIONS
>>> perhaps??
>>>
>>>
>>>
>>>
>>> Couldn't find anything similar on the Hive/LanguageManual/DDL
>>>
>>> --
>>> Kind Regards
>>>
>>>
>>>
>>> Jasper
>>>
>>
>>
>
--
Roberto Congiu -Data Engineer - OpenX
20 E Del Mar blvd, Pasadena, CA
Re: Can Hive 0.7 Rebuild partitions ?
Posted by Igor Tatarinov <ig...@decide.com>.
That's Amazon's extension to Hive and it's really handy.
On Thu, May 19, 2011 at 2:01 PM, Tim Spence <yo...@gmail.com>wrote:
> Is this functionality handled by ALTER TABLE [name] RECOVER PARTITIONS?
> Take a look at this presentation for context:
> http://www.slideshare.net/AmazonWebServices/aws-office-hours-amazon-elastic-mapreduce
>
> Best of luck,
> Tim
>
>
>
>
>
> On Thu, May 19, 2011 at 2:25 AM, Jasper Knulst <ja...@vlc.nl>wrote:
>
>> Hi,
>>
>> I have a partitioned external table on Hive 0.7. New subfolders are
>> regularly added to the base table HDFS folder.
>> I now have to perform this scan myself and let an external tool create new
>> partitions by generating and firing ALTER TABLE ADD PARTITION commands.
>>
>> Is there an easier way to have hive scan the base table folder to see if
>> there are any new partitions around? Something like REBUILD PARTITIONS
>> perhaps??
>>
>>
>>
>>
>> Couldn't find anything similar on the Hive/LanguageManual/DDL
>>
>> --
>> Kind Regards
>>
>>
>>
>> Jasper
>>
>
>
Re: Can Hive 0.7 Rebuild partitions ?
Posted by Tim Spence <yo...@gmail.com>.
Is this functionality handled by ALTER TABLE [name] RECOVER PARTITIONS?
Take a look at this presentation for context:
http://www.slideshare.net/AmazonWebServices/aws-office-hours-amazon-elastic-mapreduce
Best of luck,
Tim
On Thu, May 19, 2011 at 2:25 AM, Jasper Knulst <ja...@vlc.nl> wrote:
> Hi,
>
> I have a partitioned external table on Hive 0.7. New subfolders are
> regularly added to the base table HDFS folder.
> I now have to perform this scan myself and let an external tool create new
> partitions by generating and firing ALTER TABLE ADD PARTITION commands.
>
> Is there an easier way to have hive scan the base table folder to see if
> there are any new partitions around? Something like REBUILD PARTITIONS
> perhaps??
>
>
>
>
> Couldn't find anything similar on the Hive/LanguageManual/DDL
>
> --
> Kind Regards
>
>
>
> Jasper
>