You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Kasi Subrahmanyam <ka...@gmail.com> on 2014/03/06 11:23:27 UTC

Automatic replacement of partitions in hive

Hi,
I have a table in hive which has data of three months old. I have
partitioned the data and I got 90 partitions. Now when I get the new data
from next day I want to replace the partition 1week old with the new one
automatically.

Can this partitioning and replacement be done using swoop at the same time

Thanks,
Subbu

Re: Automatic replacement of partitions in hive

Posted by Nitin Pawar <ni...@gmail.com>.
Thanks for clarifying that Bryan


On Thu, Mar 6, 2014 at 7:55 PM, Bryan Jeffrey <br...@gmail.com>wrote:

> Nitin,
>
> #3 will not work.  msck repair table does not remove partitions if the
> files associated with the partition do not exist.  We have successfully
> applied #2 in our application.
>
> Regards,
>
> Bryan Jeffrey
>
>
> On Thu, Mar 6, 2014 at 5:37 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> There is no concept called automatic.
>>
>> Please wait for expert hive gurus to reply before using any of my
>> suggestions
>>
>> Few options which I can think of are
>> 1) Insert overwrite table with dynamic partitions enabled and restricting
>> the partition column values for the date range you want. Cost of this
>> operation will totally matter on how big the table is when you are
>> importing via sqoop.
>>
>> 2) Load data in new partition and drop older partition using hive script
>> > little bit of scripting effort is needed
>> 3) Use hadoop command line utilities to clear partition directories from
>> hdfs and then do a table repair.  I never heard anyone using this to delete
>> partition. Its mostly to recover lost partitions etc
>>
>>
>>
>>
>>
>>
>> On Thu, Mar 6, 2014 at 3:53 PM, Kasi Subrahmanyam <kasisubbu440@gmail.com
>> > wrote:
>>
>>> Hi,
>>> I have a table in hive which has data of three months old. I have
>>> partitioned the data and I got 90 partitions. Now when I get the new data
>>> from next day I want to replace the partition 1week old with the new one
>>> automatically.
>>>
>>> Can this partitioning and replacement be done using swoop at the same
>>> time
>>>
>>> Thanks,
>>> Subbu
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>


-- 
Nitin Pawar

Re: Automatic replacement of partitions in hive

Posted by Bryan Jeffrey <br...@gmail.com>.
Nitin,

#3 will not work.  msck repair table does not remove partitions if the
files associated with the partition do not exist.  We have successfully
applied #2 in our application.

Regards,

Bryan Jeffrey


On Thu, Mar 6, 2014 at 5:37 AM, Nitin Pawar <ni...@gmail.com> wrote:

> There is no concept called automatic.
>
> Please wait for expert hive gurus to reply before using any of my
> suggestions
>
> Few options which I can think of are
> 1) Insert overwrite table with dynamic partitions enabled and restricting
> the partition column values for the date range you want. Cost of this
> operation will totally matter on how big the table is when you are
> importing via sqoop.
>
> 2) Load data in new partition and drop older partition using hive script >
> little bit of scripting effort is needed
> 3) Use hadoop command line utilities to clear partition directories from
> hdfs and then do a table repair.  I never heard anyone using this to delete
> partition. Its mostly to recover lost partitions etc
>
>
>
>
>
>
> On Thu, Mar 6, 2014 at 3:53 PM, Kasi Subrahmanyam <ka...@gmail.com>wrote:
>
>> Hi,
>> I have a table in hive which has data of three months old. I have
>> partitioned the data and I got 90 partitions. Now when I get the new data
>> from next day I want to replace the partition 1week old with the new one
>> automatically.
>>
>> Can this partitioning and replacement be done using swoop at the same time
>>
>> Thanks,
>> Subbu
>>
>
>
>
> --
> Nitin Pawar
>

Re: Automatic replacement of partitions in hive

Posted by Nitin Pawar <ni...@gmail.com>.
There is no concept called automatic.

Please wait for expert hive gurus to reply before using any of my
suggestions

Few options which I can think of are
1) Insert overwrite table with dynamic partitions enabled and restricting
the partition column values for the date range you want. Cost of this
operation will totally matter on how big the table is when you are
importing via sqoop.

2) Load data in new partition and drop older partition using hive script >
little bit of scripting effort is needed
3) Use hadoop command line utilities to clear partition directories from
hdfs and then do a table repair.  I never heard anyone using this to delete
partition. Its mostly to recover lost partitions etc






On Thu, Mar 6, 2014 at 3:53 PM, Kasi Subrahmanyam <ka...@gmail.com>wrote:

> Hi,
> I have a table in hive which has data of three months old. I have
> partitioned the data and I got 90 partitions. Now when I get the new data
> from next day I want to replace the partition 1week old with the new one
> automatically.
>
> Can this partitioning and replacement be done using swoop at the same time
>
> Thanks,
> Subbu
>



-- 
Nitin Pawar