Too many hive client cause mysql meta store hang infinitely

[eric wong <wi...@gmail.com>]

Hi,

we encounter this problem in our proc hadoop cluster:

Sometimes hive mysql metastore hang because deadlock, so result in all hive
client hang infinitely.

we check in this problem, found most mysql request hang on " SHOW FULL
COLUMNS FROM `DELETEME1472008582981` FROM `hive` LIKE '%'   "

More interesting is mysql request span to multiple nodes will request table
with same name, like below:

| 5409551 | hive    | 10.93.xx.54:23327   | hive | Query       |  1476 |
Waiting for table metadata lock | SHOW FULL
COLUMNS FROM `DELETEME1472008582981` FROM `hive` LIKE '%'
               |
| 5409557 | hive    | 10.93.xx.20:3862    | hive | Query       |  1473 |
Waiting for table metadata lock | SHOW FULL
COLUMNS FROM `DELETEME1472008582981` FROM `hive` LIKE '%'
               |
| 5409567 | hive    | 10.93.xx.51:48055   | hive | Query       |  1469 |
Waiting for table metadata lock | SHOW FULL
COLUMNS FROM `DELETEME1472008582981` FROM `hive` LIKE '%'
               |
| 5409576 | hive    | 10.93.xx.32:18886   | hive | Sleep       |  1464 |
                              | NULL

               |
| 5409577 | hive    | 10.93.xx.32:18887   | hive | Query       |  1463 |
Waiting for table metadata lock | SHOW FULL
COLUMNS FROM `DELETEME1472008582981` FROM `hive` LIKE '%'
               |
| 5409587 | hive    | 10.93.25.32:19582   | hive | Query       |  1457 |
Waiting for table metadata lock | SHOW FULL
COLUMNS FROM `DELETEME1472008582981` FROM `hive` LIKE '%'

obviously it is not identical with the datanucleus code in :
RDBMSStoreManager.java
<https://github.com/datanucleus/datanucleus-rdbms/blob/master/src/main/java/org/datanucleus/store/rdbms/RDBMSStoreManager.java>,
which one hive session will create probeTable with current
milliseconds("DELETEME+"). List above shows that five hive session among
multiple node however request for ProbeTable with same name!

And we check our configuration in hive-site.xml, autoCreateSchema(false)
and fixedDataStore(true) has been set for long time. Code shows when this
two properties is set, no "DELETEMExxx" table will be created and queried.

so we want to figure out why our hive client will request for such
"DELETEMExxx" table, so result in deadlock in mysql metastore database.


Any suggestion will be welcome. Thanks!


env:

hive version: 1.2.1
metastore database: MySQL
Local/Embedded Metastore Server
Hive user: 100+


Best wishes!
-- 
Wang Haihua