You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org> on 2015/02/25 13:30:04 UTC
[jira] [Commented] (LENS-348) Lens stop is in hung state when there
is ReadTimeoutException with Hive Metastore
[ https://issues.apache.org/jira/browse/LENS-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336432#comment-14336432 ]
Amareshwari Sriramadasu commented on LENS-348:
----------------------------------------------
Seems there is a deadlock between QuerySubmitter and Shutdown, both of them trying to lock QueryExecutionServiceImpl object:
{noformat}
"QuerySubmitter" prio=10 tid=0x00007f2645db5800 nid=0x45da waiting for monitor entry [0x00007f2655ec8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.lens.server.query.QueryExecutionServiceImpl.getMetrics(QueryExecutionServiceImpl.java:307)
- waiting to lock <0x00000000c09c5f58> (a org.apache.lens.server.query.QueryExecutionServiceImpl)
at org.apache.lens.server.query.QueryExecutionServiceImpl.incrCounter(QueryExecutionServiceImpl.java:332)
at org.apache.lens.server.query.QueryExecutionServiceImpl.access$1000(QueryExecutionServiceImpl.java:71)
at org.apache.lens.server.query.QueryExecutionServiceImpl$QuerySubmitter.run(QueryExecutionServiceImpl.java:481)
at java.lang.Thread.run(Thread.java:701)
"Shutdown" prio=10 tid=0x000000000249f800 nid=0x46ae in Object.wait() [0x00007f263e6e5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c02649d0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1225)
- locked <0x00000000c02649d0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1278)
at org.apache.lens.server.query.QueryExecutionServiceImpl.stop(QueryExecutionServiceImpl.java:902)
- locked <0x00000000c09c5f58> (a org.apache.lens.server.query.QueryExecutionServiceImpl)
at org.apache.hive.service.CompositeService.stop(CompositeService.java:102)
- locked <0x00000000c0b6b858> (a org.apache.lens.server.LensServices)
at org.apache.hive.service.CompositeService.stop(CompositeService.java:92)
- locked <0x00000000c0b6b858> (a org.apache.lens.server.LensServices)
at org.apache.lens.server.LensServices.stop(LensServices.java:362)
- locked <0x00000000c0b6b858> (a org.apache.lens.server.LensServices)
at org.apache.lens.server.LensServer.stop(LensServer.java:159)
- locked <0x00000000c0b6b8b0> (a org.apache.lens.server.LensServer)
at org.apache.lens.server.LensServer$1.run(LensServer.java:259)
{noformat}
> Lens stop is in hung state when there is ReadTimeoutException with Hive Metastore
> ---------------------------------------------------------------------------------
>
> Key: LENS-348
> URL: https://issues.apache.org/jira/browse/LENS-348
> Project: Apache Lens
> Issue Type: Bug
> Reporter: Arshad Matin
> Attachments: stopIssue.txt
>
>
> LensServer.log
> {noformat}
> 25 Feb 2015 06:43:31,219 [Shutdown] INFO org.apache.lens.server.LensServices - Persistence File system object close complete
> 25 Feb 2015 06:43:31,219 [Shutdown] INFO org.apache.hive.service.AbstractService - Service:quota is stopped.
> 25 Feb 2015 06:43:31,219 [Shutdown] INFO org.apache.hive.service.AbstractService - Service:scheduler is stopped.
> 25 Feb 2015 06:43:31,219 [Shutdown] INFO org.apache.hive.service.AbstractService - Service:metastore is stopped.
> 25 Feb 2015 06:43:31,219 [Shutdown] INFO org.apache.hive.service.AbstractService - Service:query is stopped.
> 25 Feb 2015 06:43:37,609 [QuerySubmitter] ERROR org.apache.lens.server.query.QueryExecutionServiceImpl - Error launching query dacdd41b-9a39-4e3d-95fa-fe0fae93daa9
> org.apache.lens.api.LensException: org.apache.hive.service.cli.HiveSQLException: Unable to connect after 1 retries
> at org.apache.lens.driver.hive.RemoteThriftConnection.getClient(RemoteThriftConnection.java:68)
> at org.apache.lens.driver.hive.HiveDriver.getClient(HiveDriver.java:746)
> at org.apache.lens.driver.hive.HiveDriver.executeAsync(HiveDriver.java:470)
> at org.apache.lens.server.query.QueryExecutionServiceImpl$QuerySubmitter.run(QueryExecutionServiceImpl.java:464)
> at java.lang.Thread.run(Thread.java:701)
> Caused by: org.apache.hive.service.cli.HiveSQLException: Unable to connect after 1 retries
> at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:241)
> at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.newRetryingCLIServiceClient(RetryingThriftCLIServiceClient.java:227)
> at org.apache.lens.driver.hive.RemoteThriftConnection.getClient(RemoteThriftConnection.java:63)
> ... 4 more
> Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:288)
> at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connect(RetryingThriftCLIServiceClient.java:280)
> at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:237)
> ... 6 more
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:146)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> ... 12 more
> 25 Feb 2015 06:53:27,602 [HiveDriver-ConnectionExpiryThread] INFO org.apache.lens.driver.hive.HiveDriver - Closed connection:1
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)