Re: adding parquet partitions to external table (msck repair table not working)

[Vihang Karajgaonkar <vi...@cloudera.com>]

What version of Hive are you using? You should look at the HS2 logs to see
if there were any errors from msck command which ignored such partitions.
Also, would be worth to take a look at hive.msck.path.validation
configuration in case it is set to "ignore" which silently ignores
invalidate partitions. If you set it to "throw" then it would probably fail
the command explicitly which would help debugging.

On Sun, Dec 30, 2018 at 11:08 PM Colin Williams <
colin.williams.seattle@gmail.com> wrote:

> Hello,
>
> I have a parquet datasource that I'm writing via spark. For some
> reason this particular source will not pick up added partitions with
> msck repair table. However if I alter table tablename / add partition
> (key=value) then it works. However this is more cumbersome than msck
> repair table.
>
> Is there an alternative that works like msck repair table that will
> pick up the additional partitions? Otherwise I will need to fetch a
> list of partition values, and go through the process of sending sql
> requests to add each one.
>

Re: adding parquet partitions to external table (msck repair table not working)

[Colin Williams <co...@gmail.com>]

Hello,

Sorry for the late response. Just got back from vacation.

Regarding Hive version: 2.3.3-amzn-1

Regarding the HS2 logs, I don't have explicit server console access
but might be able to look at the logs and configuration with the
administrators.

I've just implemented the manual alter table / add partition steps.

Thanks,


Colin Williams


On Wed, Jan 2, 2019 at 10:17 AM Vihang Karajgaonkar <vi...@cloudera.com> wrote:
>
> What version of Hive are you using? You should look at the HS2 logs to see if there were any errors from msck command which ignored such partitions. Also, would be worth to take a look at hive.msck.path.validation configuration in case it is set to "ignore" which silently ignores invalidate partitions. If you set it to "throw" then it would probably fail the command explicitly which would help debugging.
>
> On Sun, Dec 30, 2018 at 11:08 PM Colin Williams <co...@gmail.com> wrote:
>>
>> Hello,
>>
>> I have a parquet datasource that I'm writing via spark. For some
>> reason this particular source will not pick up added partitions with
>> msck repair table. However if I alter table tablename / add partition
>> (key=value) then it works. However this is more cumbersome than msck
>> repair table.
>>
>> Is there an alternative that works like msck repair table that will
>> pick up the additional partitions? Otherwise I will need to fetch a
>> list of partition values, and go through the process of sending sql
>> requests to add each one.