[jira] [Comment Edited] (SENTRY-1630) out of sequence error in HMSFollower

["Alexander Kolbasov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067498#comment-16067498 ] 

Alexander Kolbasov edited comment on SENTRY-1630 at 6/29/17 12:20 AM:
----------------------------------------------------------------------

[~vamsee] found the root cause of this problem. The {{FullUpdateInitializer}} uses the same connection to HMS for multiple threads fetching information from HMS and this messes up the connection.

We need to have a connection pool for Hive connections instead.


was (Author: akolb):
[~vamsee] found the root cause of this problem. The FUllUpdateInitializer uses the same connection to HMS for multiple threads fetching information from HMS and this messes up the connection.

We need to have a connection pool for Hive connections instead.

> out of sequence error in HMSFollower
> ------------------------------------
>
>                 Key: SENTRY-1630
>                 URL: https://issues.apache.org/jira/browse/SENTRY-1630
>             Project: Sentry
>          Issue Type: Sub-task
>          Components: Hdfs Plugin
>    Affects Versions: sentry-ha-redesign
>            Reporter: Hao Hao
>            Priority: Blocker
>             Fix For: sentry-ha-redesign
>
>
> When HMSFollower exits in an abnormal way, for all the following connection would encounter out of sequence and SocketTimeoutException: Read timed out. Looking at HIVE-6893, it seems to relate to leakage connection problem.
> {noformat}2017-02-15 19:03:42,822 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete successfully after 0 tries. Exception got: org.apache.thrift.TApplicationException: get_database failed: out of sequence response
> 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete successfully after 0 tries. Exception got: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
> 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to metastore, current connections: 0
> 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: Exception occurred persisting Hive full snapshot into DB
> java.lang.RuntimeException: org.apache.thrift.TApplicationException: get_database failed: out of sequence response
>         at org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324)
>         at org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343)
>         at org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.TApplicationException: get_database failed: out of sequence response
>         at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84)
>         at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662)
>         at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649)
>         at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213)
>         at org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256)
>         at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110)
>         at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150)
>         at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         ... 3 more
> 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: Making a kerberos connection to HMS{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)