You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Guy Doulberg <gu...@conduit.com> on 2013/09/02 12:50:21 UTC
Hive JDBC Server: java.lang.IllegalStateException: Shutdown in progress
Hi guys,
I have a hive JDBC server in production,
It started lately to fail.
In the log files I can see the following:
2013-09-02_10:42:53.13215 java.lang.IllegalStateException: Shutdown in
progress, cannot add a shutdownHook
2013-09-02_10:42:53.13215 at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
2013-09-02_10:42:53.13216 at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2341)
2013-09-02_10:42:53.13216 at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2313)
2013-09-02_10:42:53.13217 at
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
2013-09-02_10:42:53.13217 at
org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
2013-09-02_10:42:53.13219 at
org.apache.hadoop.hive.ql.exec.Utilities.realFile(Utilities.java:1027)
2013-09-02_10:42:53.13219 at
org.apache.hadoop.hive.ql.exec.Utilities.getResourceFiles(Utilities.java:1551)
2013-09-02_10:42:53.13220 at
org.apache.hadoop.hive.ql.exec.ExecDriver.initialize(ExecDriver.java:152)
2013-09-02_10:42:53.13220 at
org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1343)
2013-09-02_10:42:53.13221 at
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
2013-09-02_10:42:53.13221 at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
2013-09-02_10:42:53.13222 at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
2013-09-02_10:42:53.13224 at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)
2013-09-02_10:42:53.13224 at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:628)
2013-09-02_10:42:53.13225 at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
2013-09-02_10:42:53.13225 at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
2013-09-02_10:42:53.13226 at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
2013-09-02_10:42:53.13226 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
2013-09-02_10:42:53.13227 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
2013-09-02_10:42:53.13227 at java.lang.Thread.run(Thread.java:662)
2013-09-02_10:42:53.13228
2013-09-02_10:42:53.13761 FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.MapRedTask
2013-09-02_10:42:53.13763 FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.MapRedTask
Has some one encoutered this problem, and know why it is happening?
The jmx doesn't expose anything interesting.
Guy
Re: Hive JDBC Server: java.lang.IllegalStateException: Shutdown in
progress
Posted by Guy Doulberg <gu...@conduit.com>.
Hi Nitin,
We are using Cdh4.2.1 for the Hadoop and for the hive,
I think that I understand the problem, when the hive process is
stopping the filesystem is closed before some of the threads.
I still need to figure out why the hive-server is restarting
On 09/02/2013 02:56 PM, Nitin Pawar wrote:
> Can you share what version of hadoop and hive are you using?
>
> This looks similar to HDFS-4841
> <https://issues.apache.org/jira/browse/HDFS-4841>
>
>
> On Mon, Sep 2, 2013 at 4:20 PM, Guy Doulberg <guy.doulberg@conduit.com
> <ma...@conduit.com>> wrote:
>
> Hi guys,
>
> I have a hive JDBC server in production,
> It started lately to fail.
>
> In the log files I can see the following:
>
> 2013-09-02_10:42:53.13215 java.lang.IllegalStateException:
> Shutdown in progress, cannot add a shutdownHook
> 2013-09-02_10:42:53.13215 at
> org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
> 2013-09-02_10:42:53.13216 at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2341)
> 2013-09-02_10:42:53.13216 at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2313)
> 2013-09-02_10:42:53.13217 at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
> 2013-09-02_10:42:53.13217 at
> org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
> 2013-09-02_10:42:53.13219 at
> org.apache.hadoop.hive.ql.exec.Utilities.realFile(Utilities.java:1027)
> 2013-09-02_10:42:53.13219 at
> org.apache.hadoop.hive.ql.exec.Utilities.getResourceFiles(Utilities.java:1551)
> 2013-09-02_10:42:53.13220 at
> org.apache.hadoop.hive.ql.exec.ExecDriver.initialize(ExecDriver.java:152)
> 2013-09-02_10:42:53.13220 at
> org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1343)
> 2013-09-02_10:42:53.13221 at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
> 2013-09-02_10:42:53.13221 at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
> 2013-09-02_10:42:53.13222 at
> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
> 2013-09-02_10:42:53.13224 at
> org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)
> 2013-09-02_10:42:53.13224 at
> org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:628)
> 2013-09-02_10:42:53.13225 at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 2013-09-02_10:42:53.13225 at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 2013-09-02_10:42:53.13226 at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> 2013-09-02_10:42:53.13226 at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 2013-09-02_10:42:53.13227 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 2013-09-02_10:42:53.13227 at
> java.lang.Thread.run(Thread.java:662)
> 2013-09-02_10:42:53.13228
> 2013-09-02_10:42:53.13761 FAILED: Execution Error, return code 3
> from org.apache.hadoop.hive.ql.exec.MapRedTask
> 2013-09-02_10:42:53.13763 FAILED: Execution Error, return code 3
> from org.apache.hadoop.hive.ql.exec.MapRedTask
>
>
> Has some one encoutered this problem, and know why it is happening?
>
> The jmx doesn't expose anything interesting.
>
>
> Guy
>
>
>
>
> --
> Nitin Pawar
Re: Hive JDBC Server: java.lang.IllegalStateException: Shutdown in progress
Posted by Nitin Pawar <ni...@gmail.com>.
Can you share what version of hadoop and hive are you using?
This looks similar to HDFS-4841<https://issues.apache.org/jira/browse/HDFS-4841>
On Mon, Sep 2, 2013 at 4:20 PM, Guy Doulberg <gu...@conduit.com>wrote:
> Hi guys,
>
> I have a hive JDBC server in production,
> It started lately to fail.
>
> In the log files I can see the following:
>
> 2013-09-02_10:42:53.13215 java.lang.**IllegalStateException: Shutdown in
> progress, cannot add a shutdownHook
> 2013-09-02_10:42:53.13215 at org.apache.hadoop.util.**
> ShutdownHookManager.**addShutdownHook(**ShutdownHookManager.java:152)
> 2013-09-02_10:42:53.13216 at org.apache.hadoop.fs.**
> FileSystem$Cache.getInternal(**FileSystem.java:2341)
> 2013-09-02_10:42:53.13216 at org.apache.hadoop.fs.**
> FileSystem$Cache.get(**FileSystem.java:2313)
> 2013-09-02_10:42:53.13217 at org.apache.hadoop.fs.**
> FileSystem.get(FileSystem.**java:351)
> 2013-09-02_10:42:53.13217 at org.apache.hadoop.fs.Path.**
> getFileSystem(Path.java:194)
> 2013-09-02_10:42:53.13219 at org.apache.hadoop.hive.ql.**
> exec.Utilities.realFile(**Utilities.java:1027)
> 2013-09-02_10:42:53.13219 at org.apache.hadoop.hive.ql.**
> exec.Utilities.**getResourceFiles(Utilities.**java:1551)
> 2013-09-02_10:42:53.13220 at org.apache.hadoop.hive.ql.**
> exec.ExecDriver.initialize(**ExecDriver.java:152)
> 2013-09-02_10:42:53.13220 at org.apache.hadoop.hive.ql.**
> Driver.launchTask(Driver.java:**1343)
> 2013-09-02_10:42:53.13221 at org.apache.hadoop.hive.ql.**
> Driver.execute(Driver.java:**1138)
> 2013-09-02_10:42:53.13221 at org.apache.hadoop.hive.ql.**
> Driver.run(Driver.java:951)
> 2013-09-02_10:42:53.13222 at org.apache.hadoop.hive.**
> service.HiveServer$**HiveServerHandler.execute(**HiveServer.java:198)
> 2013-09-02_10:42:53.13224 at org.apache.hadoop.hive.**
> service.ThriftHive$Processor$**execute.getResult(ThriftHive.**java:644)
> 2013-09-02_10:42:53.13224 at org.apache.hadoop.hive.**
> service.ThriftHive$Processor$**execute.getResult(ThriftHive.**java:628)
> 2013-09-02_10:42:53.13225 at org.apache.thrift.**
> ProcessFunction.process(**ProcessFunction.java:39)
> 2013-09-02_10:42:53.13225 at org.apache.thrift.**
> TBaseProcessor.process(**TBaseProcessor.java:39)
> 2013-09-02_10:42:53.13226 at org.apache.thrift.server.**
> TThreadPoolServer$**WorkerProcess.run(**TThreadPoolServer.java:206)
> 2013-09-02_10:42:53.13226 at java.util.concurrent.**
> ThreadPoolExecutor$Worker.**runTask(ThreadPoolExecutor.**java:886)
> 2013-09-02_10:42:53.13227 at java.util.concurrent.**
> ThreadPoolExecutor$Worker.run(**ThreadPoolExecutor.java:908)
> 2013-09-02_10:42:53.13227 at java.lang.Thread.run(Thread.**java:662)
> 2013-09-02_10:42:53.13228
> 2013-09-02_10:42:53.13761 FAILED: Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.**exec.MapRedTask
> 2013-09-02_10:42:53.13763 FAILED: Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.**exec.MapRedTask
>
>
> Has some one encoutered this problem, and know why it is happening?
>
> The jmx doesn't expose anything interesting.
>
>
> Guy
>
>
--
Nitin Pawar