[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files

["Prasanth Jayachandran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180002#comment-16180002 ] 

Prasanth Jayachandran commented on HIVE-16426:
----------------------------------------------

I don't understand how this patch handles the already running background task? When a query timeout is set, timeout monitor will set the operation state to TIMEOUT. With this patch, only the client is provided with SQLTimeoutException but the task that is actually executing on the cluster is not interrupted/cleaned up. Same is the case when user Cancels the query by ctrl + c. Isn't it?

> Query cancel: improve the way to handle files
> ---------------------------------------------
>
>                 Key: HIVE-16426
>                 URL: https://issues.apache.org/jira/browse/HIVE-16426
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Yongzhi Chen
>            Assignee: Yongzhi Chen
>             Fix For: 3.0.0
>
>         Attachments: HIVE-16426.1.patch
>
>
> 1. Add data structure support to make it is easy to check query cancel status.
> 2. Handle query cancel more gracefully. Remove possible file leaks caused by query cancel as shown in following stack:
> {noformat}
> 2017-04-11 09:57:30,727 WARN  org.apache.hadoop.hive.ql.exec.Utilities: [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories.
> java.io.InterruptedIOException: Call interrupted
>         at org.apache.hadoop.ipc.Client.call(Client.java:1496)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1439)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>         at com.sun.proxy.$Proxy20.delete(Unknown Source)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
>         at com.sun.proxy.$Proxy21.delete(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
>         at org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277)
>         at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463)
>         at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
>         at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
>         at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
>         at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
>         at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
>         at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}
> 3. Add checkpoints to related file operations to improve response time for query cancelling. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)