You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "zhihai xu (JIRA)" <ji...@apache.org> on 2016/06/08 01:12:21 UTC
[jira] [Created] (HADOOP-13247) The CACHE entry in FileSystem is
not removed if exception happened in close
zhihai xu created HADOOP-13247:
----------------------------------
Summary: The CACHE entry in FileSystem is not removed if exception happened in close
Key: HADOOP-13247
URL: https://issues.apache.org/jira/browse/HADOOP-13247
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.8.0
Reporter: zhihai xu
Assignee: zhihai xu
The CACHE entry in FileSystem is not removed if exception happened in close. It causes "Filesystem closed" IOException if the same filesystem is used later.
The following is stack trace for the exception coming out of close:
{code}
2016-06-07 18:21:18,201 ERROR hive.ql.exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:756)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4022)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:172)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1679)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1422)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1205)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1052)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1047)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:158)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:76)
at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:219)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1383)
at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2006)
at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:900)
at org.apache.hadoop.hive.metastore.Warehouse.closeFs(Warehouse.java:122)
at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:497)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.createTempTable(SessionHiveMetaStoreClient.java:345)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:664)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:652)
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy8.createTable(Unknown Source)
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1909)
at com.sun.proxy.$Proxy8.createTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:750)
... 21 more
Caused by: java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:151)
... 48 more
{code}
After this, the old DistributedFileSystem is still stored in the FileSystem CACHE although DFSClient in DistributedFileSystem is closed.
This caused any later DistributedFileSystem operation failed with "Filesystem closed" IOException:
{code}
2016-06-07 18:21:19,024 WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error opening session:
org.apache.hive.service.cli.HiveSQLException: Failed to open new session: java.lang.RuntimeException: java.io.IOException: Filesystem closed
at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:288)
at org.apache.hive.service.cli.CLIService.openSession(CLIService.java:178)
at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:428)
at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:319)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.IOException: Filesystem closed
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:511)
at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:138)
at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:280)
... 12 more
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1986)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:587)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:545)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:497)
... 14 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org