dynamic partitioning issues in hive 0.13.0

[Viral Bajaria <vi...@gmail.com>]

Hi,

I have been running into 2 issues with dynamic partitioning queries.

The query runs fine and I get errors like:

*Failed with exception
MetaException(message:javax.jdo.JDODataStoreException: Insert of object
"org.apache.hadoop.hive.metastore.model.MPartition@5bbd373e" using
statement "INSERT INTO `PARTITIONS`
(`PART_ID`,`LAST_ACCESS_TIME`,`TBL_ID`,`SD_ID`,`CREATE_TIME`,`PART_NAME`)
VALUES (?,?,?,?,?,?)" failed : Deadlock found when trying to get lock; try
restarting transaction*

Detailed exception: https://gist.github.com/vbajaria/aead8eb4597f31df4807

Why would there be a deadlock on adding a new partition info to the
metastore ? It seems weird that would even happen given that the partition
did not even exist prior to this query run.

I am thinking it could be an issue because I have 100s of such queries
running and the INDEX on that table might be causing the issue ?


The 2nd issue is more painful. After the 1st error mentioned above, I have
a process which figures out what failed and then re-runs the data for that
month. After the query finishes successfully, it ends up adding only 1
partition instead of partition for each day in that month. If I run the
query manually and look at the data returned then it returns data for all
partitions, yet for some reason dynamic partition does not ad all. The
query is an "INSERT OVERWRITE" query.

In the end I have to drop all partitions for that month before re-running
the query and then everything works fine. Without that the partitions that
were missed on first attempt don't show up.

The missing partitions are in itself a mystery.

Any points on how to debug this ?

Thanks,
Viral

Re: dynamic partitioning issues in hive 0.13.0

[Viral Bajaria <vi...@gmail.com>]

What should I be checking for ?

I have logs in the system but unsure what should I look for, any pointers ?

The errors happen so quickly that I am not able to trap the calls and look
at the mysql side on what's happening.

With regards to the query not loading data into a partition even though
there is data, you think that is also a metastore issue ?

Thanks,
Viral


On Mon, Dec 1, 2014 at 6:11 PM, Nagarjuna Vissarapu <
nagarjuna.viss@gmail.com> wrote:

> Can you please check you mysql service which stores metada
> On Dec 2, 2014 4:18 AM, "Viral Bajaria" <vi...@gmail.com> wrote:
>
>> Hi,
>>
>> I have been running into 2 issues with dynamic partitioning queries.
>>
>> The query runs fine and I get errors like:
>>
>> *Failed with exception
>> MetaException(message:javax.jdo.JDODataStoreException: Insert of object
>> "org.apache.hadoop.hive.metastore.model.MPartition@5bbd373e" using
>> statement "INSERT INTO `PARTITIONS`
>> (`PART_ID`,`LAST_ACCESS_TIME`,`TBL_ID`,`SD_ID`,`CREATE_TIME`,`PART_NAME`)
>> VALUES (?,?,?,?,?,?)" failed : Deadlock found when trying to get lock; try
>> restarting transaction*
>>
>> Detailed exception: https://gist.github.com/vbajaria/aead8eb4597f31df4807
>>
>> Why would there be a deadlock on adding a new partition info to the
>> metastore ? It seems weird that would even happen given that the partition
>> did not even exist prior to this query run.
>>
>> I am thinking it could be an issue because I have 100s of such queries
>> running and the INDEX on that table might be causing the issue ?
>>
>>
>> The 2nd issue is more painful. After the 1st error mentioned above, I
>> have a process which figures out what failed and then re-runs the data for
>> that month. After the query finishes successfully, it ends up adding only 1
>> partition instead of partition for each day in that month. If I run the
>> query manually and look at the data returned then it returns data for all
>> partitions, yet for some reason dynamic partition does not ad all. The
>> query is an "INSERT OVERWRITE" query.
>>
>> In the end I have to drop all partitions for that month before re-running
>> the query and then everything works fine. Without that the partitions that
>> were missed on first attempt don't show up.
>>
>> The missing partitions are in itself a mystery.
>>
>> Any points on how to debug this ?
>>
>> Thanks,
>> Viral
>>
>>
>>

Re: dynamic partitioning issues in hive 0.13.0

[Nagarjuna Vissarapu <na...@gmail.com>]

Can you please check you mysql service which stores metada
On Dec 2, 2014 4:18 AM, "Viral Bajaria" <vi...@gmail.com> wrote:

> Hi,
>
> I have been running into 2 issues with dynamic partitioning queries.
>
> The query runs fine and I get errors like:
>
> *Failed with exception
> MetaException(message:javax.jdo.JDODataStoreException: Insert of object
> "org.apache.hadoop.hive.metastore.model.MPartition@5bbd373e" using
> statement "INSERT INTO `PARTITIONS`
> (`PART_ID`,`LAST_ACCESS_TIME`,`TBL_ID`,`SD_ID`,`CREATE_TIME`,`PART_NAME`)
> VALUES (?,?,?,?,?,?)" failed : Deadlock found when trying to get lock; try
> restarting transaction*
>
> Detailed exception: https://gist.github.com/vbajaria/aead8eb4597f31df4807
>
> Why would there be a deadlock on adding a new partition info to the
> metastore ? It seems weird that would even happen given that the partition
> did not even exist prior to this query run.
>
> I am thinking it could be an issue because I have 100s of such queries
> running and the INDEX on that table might be causing the issue ?
>
>
> The 2nd issue is more painful. After the 1st error mentioned above, I have
> a process which figures out what failed and then re-runs the data for that
> month. After the query finishes successfully, it ends up adding only 1
> partition instead of partition for each day in that month. If I run the
> query manually and look at the data returned then it returns data for all
> partitions, yet for some reason dynamic partition does not ad all. The
> query is an "INSERT OVERWRITE" query.
>
> In the end I have to drop all partitions for that month before re-running
> the query and then everything works fine. Without that the partitions that
> were missed on first attempt don't show up.
>
> The missing partitions are in itself a mystery.
>
> Any points on how to debug this ?
>
> Thanks,
> Viral
>
>
>