You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by "Krishnappa, Suresh" <Su...@rsa.com> on 2013/02/07 20:46:53 UTC

msck repair table not adding partitions which contains data.

Hi All,
I have created a partitioned HIVE external table as follows

create external table test_part (key int, val int) partitioned by (part int) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/test/';

I have the following folders and files in /test

/test/part=1
/test/part=1/1.txt
/test/part=2
/test/part=2/1.txt
/test/part=3
/test/part=4

Now I try to automatically add the partitions into the table using the 'msck repair' command
hive> msck repair table test_part;
OK
Partitions not in metastore:    test_part:part=3        test_part:part=4
Repair: Added partition to metastore test_part:part=3
Repair: Added partition to metastore test_part:part=4
Time taken: 0.685 seconds

As you can see only partitions which do not contain any data have been added. part=1 and part=2 folders have been ignored.
Is this by design? Am I using this correctly?

The only alternative I found is to explicitly add partitions using 'alter table add partition' once every subfolder.
Is there a simpler way to achieve this?

Thanks in advance
Suri

RE: msck repair table not adding partitions which contains data.

Posted by "Krishnappa, Suresh" <Su...@rsa.com>.

Thanks Mark

From: Mark Grover [mailto:grover.markgrover@gmail.com]
Sent: Thursday, February 07, 2013 2:54 PM
To: user@hive.apache.org
Subject: Re: msck repair table not adding partitions which contains data.

Suresh,
Take a look at this:
https://issues.apache.org/jira/browse/HIVE-3231
On Thu, Feb 7, 2013 at 11:46 AM, Krishnappa, Suresh <Su...@rsa.com>> wrote:
Hi All,
I have created a partitioned HIVE external table as follows

create external table test_part (key int, val int) partitioned by (part int) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/test/';

I have the following folders and files in /test

/test/part=1
/test/part=1/1.txt
/test/part=2
/test/part=2/1.txt
/test/part=3
/test/part=4

Now I try to automatically add the partitions into the table using the 'msck repair' command
hive> msck repair table test_part;
OK
Partitions not in metastore:    test_part:part=3        test_part:part=4
Repair: Added partition to metastore test_part:part=3
Repair: Added partition to metastore test_part:part=4
Time taken: 0.685 seconds

As you can see only partitions which do not contain any data have been added. part=1 and part=2 folders have been ignored.
Is this by design? Am I using this correctly?

The only alternative I found is to explicitly add partitions using 'alter table add partition' once every subfolder.
Is there a simpler way to achieve this?

Thanks in advance
Suri

Re: msck repair table not adding partitions which contains data.

Posted by Mark Grover <gr...@gmail.com>.

Suresh,
Take a look at this:
https://issues.apache.org/jira/browse/HIVE-3231

On Thu, Feb 7, 2013 at 11:46 AM, Krishnappa, Suresh <
Suresh.Krishnappa@rsa.com> wrote:

> Hi All,****
>
> I have created a partitioned HIVE external table as follows****
>
> ** **
>
> create external table test_part (key int, val int) partitioned by (part
> int) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE LOCATION '/test/';****
>
> ** **
>
> I have the following folders and files in /test****
>
> ** **
>
> /test/part=1****
>
> /test/part=1/1.txt****
>
> /test/part=2****
>
> /test/part=2/1.txt****
>
> /test/part=3****
>
> /test/part=4****
>
> ** **
>
> Now I try to automatically add the partitions into the table using the
> 'msck repair' command****
>
> hive> msck repair table test_part;****
>
> *OK*
>
> *Partitions not in metastore:    test_part:part=3        test_part:part=4*
>
> *Repair: Added partition to metastore test_part:part=3*
>
> *Repair: Added partition to metastore test_part:part=4*
>
> *Time taken: 0.685 seconds*
>
> ** **
>
> As you can see only partitions which do not contain any data have been
> added. part=1 and part=2 folders have been ignored.****
>
> Is this by design? Am I using this correctly?****
>
> ** **
>
> The only alternative I found is to explicitly add partitions using 'alter
> table add partition' once every subfolder. ****
>
> Is there a simpler way to achieve this?****
>
> ** **
>
> Thanks in advance****
>
> Suri****
>