You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jakub Holy <ja...@iterate.no> on 2013/02/08 13:04:29 UTC
Why does "HiveException: Unable to fetch table XYZ" happen?
Hi everybody, I hope you'll be able to help me understand why our Hive
server occasionally starts throwing "HiveException: Unable to fetch table
<same table name>" and subsequently noe JDBC calls get through.
I have observed this mostly after having started the server so I guess
there could be some timing issue in Hive initialization. Most of the time
Hive works all right so there no defect in our configuration/infrastructure
per se.
Any idea how this could have been happening and how to prevent/fix it? We
use Hive 0.9.0 and connect to it via JDBC over Thrift. When I restart Hive,
the problem disappears.
Thanks a lot!
Here is the full log with some context around:
2013-02-08 09:28:43,943 WARN mapred.JobClient
(JobClient.java:copyAndConfigureFiles(667)) - Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
2013-02-08 09:28:53,399 ERROR ql.Driver (SessionState.java:printError(400))
- FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2013-02-08 09:29:51,997 ERROR parse.SemanticAnalyzer
(SemanticAnalyzer.java:getMetaData(1128)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table
playerevents
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:896)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:830)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:954)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7524)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:629)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:617)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.jdo.JDOException: Attempt to instantiate insert request
gave an error
NestedThrowables:
java.lang.reflect.InvocationTargetException
at
org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:434)
at
org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:539)
at
org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551)
at
org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:800)
at
org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:733)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
at $Proxy0.getTable(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:974)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:735)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:888)
... 16 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:588)
at
org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:324)
at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.getFetchRequest(RDBMSPersistenceHandler.java:267)
at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:239)
at
org.datanucleus.jdo.state.JDOStateManagerImpl.loadFieldsFromDatastore(JDOStateManagerImpl.java:1929)
at
org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields(JDOStateManagerImpl.java:1597)
at
org.datanucleus.jdo.state.Hollow.transitionRetrieve(Hollow.java:168)
at
org.datanucleus.state.AbstractStateManager.retrieve(AbstractStateManager.java:470)
at
org.datanucleus.ObjectManagerImpl.retrieveObject(ObjectManagerImpl.java:1131)
at
org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:534)
... 28 more
Caused by: java.lang.IllegalStateException: Table object has not been been
initialised : `SDS`
at
org.datanucleus.store.rdbms.table.AbstractTable.assertIsInitialized(AbstractTable.java:625)
at
org.datanucleus.store.rdbms.table.ClassTable.getMemberMapping(ClassTable.java:2904)
at
org.datanucleus.store.rdbms.sql.SQLStatementHelper.selectMemberOfSourceInStatement(SQLStatementHelper.java:512)
at
org.datanucleus.store.rdbms.sql.SQLStatementHelper.selectFetchPlanOfSourceClassInStatement(SQLStatementHelper.java:441)
at
org.datanucleus.store.rdbms.sql.SQLStatementHelper.selectMemberOfSourceInStatement(SQLStatementHelper.java:567)
at
org.datanucleus.store.rdbms.request.FetchRequest.processMembersOfClass(FetchRequest.java:468)
at
org.datanucleus.store.rdbms.request.FetchRequest.<init>(FetchRequest.java:166)
... 42 more
2013-02-08 09:29:52,016 ERROR ql.Driver (SessionState.java:printError(400))
- FAILED: Error in semantic analysis: Unable to fetch table playerevents
org.apache.hadoop.hive.ql.parse.SemanticException: Unable to fetch table
playerevents
--
*A world without pair-programming is a sad, lonely place.*
*
(Vær så snill og hjelp meg med å korrigere min norsk, skriftlig og muntlig.
Takk!)*
Jakub Holy
Solutions Engineer | +47 966 23 666
Iterate AS | www.iterate.no
The Lean Software Development Consultancy
- http://theholyjava.wordpress.com/ -
Re: Why does "HiveException: Unable to fetch table XYZ" happen?
Posted by Edward Capriolo <ed...@gmail.com>.
In short, hiveserver has leaks, the upcoming hiveserver2 (and the
changes made in hive trunk due to it should prevent many of these. We
run our hiveserver's behind a load balancer (4 of them) basically this
increases the mean time to failure due to leaks. It would be good if
hive-server had some jmx or some method to determine # of connected
clients and then an admin can script a restart in a dead time.
IF you are using zk locking turn that off, zklocking+hiveserver leaks
much more then just plan hiveserver.
On Fri, Feb 8, 2013 at 7:04 AM, Jakub Holy <ja...@iterate.no> wrote:
> Hi everybody, I hope you'll be able to help me understand why our Hive
> server occasionally starts throwing "HiveException: Unable to fetch table
> <same table name>" and subsequently noe JDBC calls get through.
>
> I have observed this mostly after having started the server so I guess there
> could be some timing issue in Hive initialization. Most of the time Hive
> works all right so there no defect in our configuration/infrastructure per
> se.
>
> Any idea how this could have been happening and how to prevent/fix it? We
> use Hive 0.9.0 and connect to it via JDBC over Thrift. When I restart Hive,
> the problem disappears.
>
> Thanks a lot!
>
> Here is the full log with some context around:
>
> 2013-02-08 09:28:43,943 WARN mapred.JobClient
> (JobClient.java:copyAndConfigureFiles(667)) - Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 2013-02-08 09:28:53,399 ERROR ql.Driver (SessionState.java:printError(400))
> - FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> 2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2013-02-08 09:29:51,997 ERROR parse.SemanticAnalyzer
> (SemanticAnalyzer.java:getMetaData(1128)) -
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table
> playerevents
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:896)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:830)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:954)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7524)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
> at
> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191)
> at
> org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:629)
> at
> org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:617)
> at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: javax.jdo.JDOException: Attempt to instantiate insert request
> gave an error
> NestedThrowables:
> java.lang.reflect.InvocationTargetException
> at
> org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:434)
> at
> org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:539)
> at
> org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551)
> at
> org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:800)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:733)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
> at $Proxy0.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:974)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:735)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:888)
> ... 16 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:588)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:324)
> at
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.getFetchRequest(RDBMSPersistenceHandler.java:267)
> at
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:239)
> at
> org.datanucleus.jdo.state.JDOStateManagerImpl.loadFieldsFromDatastore(JDOStateManagerImpl.java:1929)
> at
> org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields(JDOStateManagerImpl.java:1597)
> at
> org.datanucleus.jdo.state.Hollow.transitionRetrieve(Hollow.java:168)
> at
> org.datanucleus.state.AbstractStateManager.retrieve(AbstractStateManager.java:470)
> at
> org.datanucleus.ObjectManagerImpl.retrieveObject(ObjectManagerImpl.java:1131)
> at
> org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:534)
> ... 28 more
> Caused by: java.lang.IllegalStateException: Table object has not been been
> initialised : `SDS`
> at
> org.datanucleus.store.rdbms.table.AbstractTable.assertIsInitialized(AbstractTable.java:625)
> at
> org.datanucleus.store.rdbms.table.ClassTable.getMemberMapping(ClassTable.java:2904)
> at
> org.datanucleus.store.rdbms.sql.SQLStatementHelper.selectMemberOfSourceInStatement(SQLStatementHelper.java:512)
> at
> org.datanucleus.store.rdbms.sql.SQLStatementHelper.selectFetchPlanOfSourceClassInStatement(SQLStatementHelper.java:441)
> at
> org.datanucleus.store.rdbms.sql.SQLStatementHelper.selectMemberOfSourceInStatement(SQLStatementHelper.java:567)
> at
> org.datanucleus.store.rdbms.request.FetchRequest.processMembersOfClass(FetchRequest.java:468)
> at
> org.datanucleus.store.rdbms.request.FetchRequest.<init>(FetchRequest.java:166)
> ... 42 more
> 2013-02-08 09:29:52,016 ERROR ql.Driver (SessionState.java:printError(400))
> - FAILED: Error in semantic analysis: Unable to fetch table playerevents
> org.apache.hadoop.hive.ql.parse.SemanticException: Unable to fetch table
> playerevents
> --
> A world without pair-programming is a sad, lonely place.
>
> (Vær så snill og hjelp meg med å korrigere min norsk, skriftlig og muntlig.
> Takk!)
>
> Jakub Holy
> Solutions Engineer | +47 966 23 666
> Iterate AS | www.iterate.no
> The Lean Software Development Consultancy
> - http://theholyjava.wordpress.com/ -