You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org> on 2016/03/21 10:32:25 UTC
[jira] [Commented] (LENS-893) Query Service should not mark queries
failed in case driver is not responding in a timely manner
[ https://issues.apache.org/jira/browse/LENS-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203940#comment-15203940 ]
Amareshwari Sriramadasu commented on LENS-893:
----------------------------------------------
We are seeing following exceptions in our production:
{noformat}
17 Mar 2016 08:58:58 [c6000f1d-f098-4b94-8298-36fcea70edf6] [StatusPoller] ERROR org.apache.lens.server.query.QueryExecutionServiceImpl - Status update failed for c6000f1d-f098-4b94-8298-36fcea70edf6
org.apache.lens.server.api.error.LensException: Internal Server Error.
at org.apache.lens.driver.hive.HiveDriver.updateStatus(HiveDriver.java:681) ~[lens-driver-hive-2.6.0-inm.jar:2.6.0-inm]
at org.apache.lens.server.query.QueryExecutionServiceImpl.updateStatus(QueryExecutionServiceImpl.java:887) [classes/:na]
at org.apache.lens.server.query.QueryExecutionServiceImpl.access$1600(QueryExecutionServiceImpl.java:98) [classes/:na]
at org.apache.lens.server.query.QueryExecutionServiceImpl$StatusPoller.run(QueryExecutionServiceImpl.java:789) [classes/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: org.apache.hive.service.cli.HiveSQLException: Unable to connect after 3 retries
at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:241) ~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.invoke(RetryingThriftCLIServiceClient.java:335) ~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at com.sun.proxy.$Proxy44.getOperationStatus(Unknown Source) ~[na:na]
at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient$CLIServiceClientWrapper.getOperationStatus(RetryingThriftCLIServiceClient.java:162) ~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at org.apache.lens.driver.hive.HiveDriver.updateStatus(HiveDriver.java:601) ~[lens-driver-hive-2.6.0-inm.jar:2.6.0-inm]
... 4 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Socket closed
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:288) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connect(RetryingThriftCLIServiceClient.java:280) ~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:237) ~[hive-service-0.13.4-inm.jar:0.13.4-inm]
... 8 common frames omitted
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:203) ~[na:1.8.0_72]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_72]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_72]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[na:1.8.0_72]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_72]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
... 14 common frames omitted
{noformat}
or
{noformat}
17 Mar 2016 08:49:55 [c6000f1d-f098-4b94-8298-36fcea70edf6] [StatusPoller] ERROR org.apache.lens.driver.hive.HiveDriver - Error getting query status
org.apache.hive.service.cli.HiveSQLException: Unable to connect after 3 retries
at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:241) ~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.invoke(RetryingThriftCLIServiceClient.java:335) ~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at com.sun.proxy.$Proxy44.getOperationStatus(Unknown Source) ~[na:na]
at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient$CLIServiceClientWrapper.getOperationStatus(RetryingThriftCLIServiceClient.java:162) ~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at org.apache.lens.driver.hive.HiveDriver.updateStatus(HiveDriver.java:601) ~[lens-driver-hive-2.6.0-inm.jar:2.6.0-inm]
at org.apache.lens.server.query.QueryExecutionServiceImpl.updateStatus(QueryExecutionServiceImpl.java:887) [classes/:na]
at org.apache.lens.server.query.QueryExecutionServiceImpl.access$1600(QueryExecutionServiceImpl.java:98) [classes/:na]
at org.apache.lens.server.query.QueryExecutionServiceImpl$StatusPoller.run(QueryExecutionServiceImpl.java:789) [classes/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Socket closed
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:288) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connect(RetryingThriftCLIServiceClient.java:280) ~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:237) ~[hive-service-0.13.4-inm.jar:0.13.4-inm]
... 8 common frames omitted
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:203) ~[na:1.8.0_72]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_72]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_72]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[na:1.8.0_72]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_72]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
... 14 common frames omitted
{noformat}
Planning to add exponential back-off for next retry from lens status poller, and getStatus API call would give last known status.
> Query Service should not mark queries failed in case driver is not responding in a timely manner
> ------------------------------------------------------------------------------------------------
>
> Key: LENS-893
> URL: https://issues.apache.org/jira/browse/LENS-893
> Project: Apache Lens
> Issue Type: Improvement
> Components: server
> Reporter: Rajat Khandelwal
> Assignee: Amareshwari Sriramadasu
> Fix For: 2.6
>
>
> Exponential backoff can be done in update status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)