[jira] [Created] (HIVE-24507) "File file:XXX.jar does not exist" when changing content of "hive.reloadable.aux.jars.path" directory content

["Arnaud Linz (Jira)" <ji...@apache.org>]

Arnaud Linz created HIVE-24507:
----------------------------------

             Summary: "File file:XXX.jar does not exist" when changing content of "hive.reloadable.aux.jars.path" directory content
                 Key: HIVE-24507
                 URL: https://issues.apache.org/jira/browse/HIVE-24507
             Project: Hive
          Issue Type: Bug
          Components: HiveServer2
    Affects Versions: 2.1.1
            Reporter: Arnaud Linz


Purpose of hive.reloadable.aux.jars.path, introduced by https://issues.apache.org/jira/browse/HIVE-7553 was do avoid scheduling maintenance window for every jar change, but it is not enough.

On a large system, the lack of atomicity between the directory listing of jars contained in hive.reloadable.aux.jars.path and the actual use of the file when uploaded to the job's yarn resources lead to query failures, even if no jar/UDF is used in the failing query (because it is a global parameter).

Stack trace sample:
{code:java}
 File file:/XXX.jar does not exist   at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)   at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)   at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)   at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:378)   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329)   at org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:703)   at org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:315)   at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:207)   at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:135)   at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)   at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)   at java.security.AccessController.doPrivileged(Native Method)   at javax.security.auth.Subject.doAs(Subject.java:422)   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)   at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)   at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571)   at java.security.AccessController.doPrivileged(Native Method)   at javax.security.auth.Subject.doAs(Subject.java:422)   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)   at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571)   at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562)   at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:444)   at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)   at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200)   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843)   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563)   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1334)   at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)   at org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92)   at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)   at java.security.AccessController.doPrivileged(Native Method)   at javax.security.auth.Subject.doAs(Subject.java:422)   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)   at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)   at java.util.concurrent.FutureTask.run(FutureTask.java:266)   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)   at java.lang.Thread.run(Thread.java:748){code}
 

It's probably not possible to achieve atomicity, but this lack of atomicity should be taken into account and this error should be a warning. Actually, if a jar is removed, it's probably because no query are using it any longer. And if it was really used, it will trigger another ClassNotFound error later that, with the warning log, can suffice.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)