You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Sergey Shelukhin (JIRA)" <ji...@apache.org> on 2015/12/05 23:34:10 UTC
[jira] [Comment Edited] (HIVE-12602) Enabling
hive.orc.splits.ms.footer.cache.enabled=true throws exceptions
[ https://issues.apache.org/jira/browse/HIVE-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15043490#comment-15043490 ]
Sergey Shelukhin edited comment on HIVE-12602 at 12/5/15 10:33 PM:
-------------------------------------------------------------------
This feature is not supported on non-HBase metastore at all. If there's embedded HBase metastore it should still work thru Hive.
Moreover, if ObjectStore is able to initialize, it will work fine, the calls will just do nothing.
I think this is not an issue... probably a comment for a setting could be expanded to say that first, for now it's only supported on HBase metastore, and then, if this is set to true, AM must be able to init the metastore (addendum: if local metastore is used, or connect if remote metastore is used), i.e. jars need to be shipped. That would cover all cases and so this becomes a user error.
was (Author: sershe):
This feature is not supported on non-HBase metastore at all. If there's embedded HBase metastore it should still work thru Hive.
Moreover, if ObjectStore is able to initialize, it will work fine, the calls will just do nothing.
I think this is not an issue... probably a comment for a setting could be expanded to say that first, for now it's only supported on HBase metastore, and then, if this is set to true, AM must be able to init the metastore, i.e. jars need to be shipped. That would cover all cases and so this becomes a user error.
> Enabling hive.orc.splits.ms.footer.cache.enabled=true throws exceptions
> -----------------------------------------------------------------------
>
> Key: HIVE-12602
> URL: https://issues.apache.org/jira/browse/HIVE-12602
> Project: Hive
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Attachments: HIVE-12602.WIP.patch
>
>
> Env: master branch
> {noformat}
> Vertex failed, vertexName=Map 4, vertexId=vertex_1448526568607_12618_1_00, diagnostics=[Vertex vertex_1448526568607_12618_1_00 [Map 4] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: at initializer failed, vertex=vertex_1448526568607_12618_1_00 [Map 4], java.lang.RuntimeException: ORC split generation failed with exception: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1272)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1338)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:366)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:478)
> at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:183)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1266)
> ... 15 more
> Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1555)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
> at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3055)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3075)
> at org.apache.hadoop.hive.ql.metadata.Hive.putFileMetadata(Hive.java:3382)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$MetastoreCache.put(OrcInputFormat.java:1913)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1143)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1046)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:903)
> ... 4 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1553)
> ... 14 more
> Caused by: javax.jdo.JDOFatalUserException: Class org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
> Caused by: javax.jdo.JDOFatalUserException: Class org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
> NestedThrowables:
> java.lang.ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory
> at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1175)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
> at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:409)
> at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:438)
> at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:332)
> at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:288)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:55)
> at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:64)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:516)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:481)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:547)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:370)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
> at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5749)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:219)
> at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:67)
> ... 19 more
> Caused by: java.lang.ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at javax.jdo.JDOHelper$18.run(JDOHelper.java:2018)
> at javax.jdo.JDOHelper$18.run(JDOHelper.java:2016)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.forName(JDOHelper.java:2015)
> at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1162)
> ... 38 more
> ]
> {noformat}
> Ran a query by passing "--hiveconf hive.orc.splits.ms.footer.cache.enabled=true" in hive cli. First time encountered this exception, when the same query got executed second in the same session, it worked fine without exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)