[jira] [Commented] (HIVE-6893) out of sequence error in HiveMetastore server

["Naveen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969572#comment-13969572 ] 

Naveen commented on HIVE-6893:
------------------------------

Mestastore is running as a standalone process/service outside the HS2 JVM. This issue was seen after the following occured (there is no indication of a problem prior)
1) Upgrade the HMS DB (mysql)
2) Running with slightly newer version of HUE.
3) About 15 parallel beeswax sessions to HS2 each running a query.

We haven't been able to reproduce this in-house using a JDBC Client that spins off threads that connect to HS2, run a query & exit. 

We suspected that it was one of the two causes
1) Socket timeouts during read by parallel invocations between HS2 (HMS Client) & HMS. 
2) Concurrency issues in HMS on invocations to HMS.

After noticing Socket timeout exceptions in the HS2 logs, we had the customer increase the socket read timeouts from 90s to 120s. Did not help, but also the hive logs indicate that the "out of sequence" errors occurred first then the socket  timeouts, pretty far apart in time. So the socket read timeout is not likely the root cause.

It must be concurrent access to the HMS by multiple beeswax sessions thats causing this issue. 

Hope this helps.

> out of sequence error in HiveMetastore server
> ---------------------------------------------
>
>                 Key: HIVE-6893
>                 URL: https://issues.apache.org/jira/browse/HIVE-6893
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 0.12.0
>            Reporter: Romain Rigaux
>
> Calls listing databases or tables fail. It seems to be a concurrency problem.
> {code}
> 014-03-06 05:34:00,785 ERROR hive.log: org.apache.thrift.TApplicationException: get_databases failed: out of sequence response
>         at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
>         at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
>         at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
>         at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
>         at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
>         at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
>         at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>         at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
>         at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
>         at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
>         at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
>         at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
>         at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
>         at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
>         at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>         at org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
>         at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)