You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Eric Chu <ec...@rocketfuel.com> on 2013/08/07 01:39:40 UTC
DataNucleus patches for Hive
Hi,
I'm a bit confused about what DataNucleus patches we should get for *Hive
0.11 with JDK 6*. It'd be great if people working on that could shed some
light on the subject. Thanks in advance!
After installing Hive 0.11 and applying the patch for HIVE-4619 (or else MR
queries will result in error), I will sometimes (but not always) see
DataNucleus error log messages as below:
13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
requires "org.eclipse.core.resources" but it cannot be resolved.
13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
requires "org.eclipse.core.runtime" but it cannot be resolved.
13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
requires "org.eclipse.text" but it cannot be resolved.
HIVE-2015 was opened for this issue, but the last comment there said it was
fixed through HIVE-3632, whose purpose is to upgrade DataNucleus to support
JDK 7. HIVE-3632 in turn was blocked by HIVE-4900 and broke HIVE-4942, both
of which have patches available in 12. So I ended up applying HIVE-4900,
HIVE-3632, and HIVE-4942, in that order. However, afterward, any query (MR,
select *, show tables) would result in error.
In the end, after I applied the patch for HIVE-4900, I haven't seen those
error messages and queries are still running fine. Are the other patches
not compatible with JDK 6?
*Error messages for SHOW TABLES after applying HIVE-4900, HIVE-3632, and
HIVE-4942:*
FAILED: Error in metadata: java.lang.RuntimeException: Unable to
instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
13/08/06 12:31:01 ERROR exec.Task: FAILED: Error in metadata:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1140)
at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1125)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2265)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:337)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1361)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1146)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:948)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:261)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:218)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:758)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1139)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2284)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2295)
at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1136)
... 18 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1137)
... 23 more
Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
at
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:262)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:291)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:224)
at
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:199)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:62)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:413)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:401)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:439)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:325)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:285)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:53)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:58)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4102)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
... 28 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
at
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
... 47 more
Caused by: java.lang.NoSuchFieldError: NUCLEUS_CONTEXT_LOADER
at org.datanucleus.NucleusContext.<clinit>(NucleusContext.java:105)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:395)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:294)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
... 55 more
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
13/08/06 12:31:01 ERROR ql.Driver: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLTask
Re: DataNucleus patches for Hive
Posted by Eric Chu <ec...@rocketfuel.com>.
Xuefu,
Indeed that's the problem and "ant very-clean" also fixed the problem.
Thanks,
Eric
On Tue, Aug 6, 2013 at 5:10 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
> You probably have mix-matched datanucleus libraries in your lib/ folder.
> You should only three 3.2.x files if you do "ls datanu*". To avoid that,
> you need to do "ant very-clean" before building hive.
>
> --Xuefu
>
>
> On Tue, Aug 6, 2013 at 4:39 PM, Eric Chu <ec...@rocketfuel.com> wrote:
>
> > Hi,
> >
> > I'm a bit confused about what DataNucleus patches we should get for *Hive
> > 0.11 with JDK 6*. It'd be great if people working on that could shed some
> > light on the subject. Thanks in advance!
> >
> > After installing Hive 0.11 and applying the patch for HIVE-4619 (or else
> MR
> > queries will result in error), I will sometimes (but not always) see
> > DataNucleus error log messages as below:
> >
> > 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
> > requires "org.eclipse.core.resources" but it cannot be resolved.
> > 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
> > requires "org.eclipse.core.runtime" but it cannot be resolved.
> > 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
> > requires "org.eclipse.text" but it cannot be resolved.
> >
> > HIVE-2015 was opened for this issue, but the last comment there said it
> was
> > fixed through HIVE-3632, whose purpose is to upgrade DataNucleus to
> support
> > JDK 7. HIVE-3632 in turn was blocked by HIVE-4900 and broke HIVE-4942,
> both
> > of which have patches available in 12. So I ended up applying HIVE-4900,
> > HIVE-3632, and HIVE-4942, in that order. However, afterward, any query
> (MR,
> > select *, show tables) would result in error.
> >
> > In the end, after I applied the patch for HIVE-4900, I haven't seen those
> > error messages and queries are still running fine. Are the other patches
> > not compatible with JDK 6?
> >
> > *Error messages for SHOW TABLES after applying HIVE-4900, HIVE-3632, and
> > HIVE-4942:*
> >
> > FAILED: Error in metadata: java.lang.RuntimeException: Unable to
> > instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> > 13/08/06 12:31:01 ERROR exec.Task: FAILED: Error in metadata:
> > java.lang.RuntimeException: Unable to instantiate
> > org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> > org.apache.hadoop.hive.ql.metadata.HiveException:
> > java.lang.RuntimeException: Unable to instantiate
> > org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1140)
> > at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1125)
> > at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2265)
> > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:337)
> > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)
> > at
> >
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1361)
> > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1146)
> > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:948)
> > at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:261)
> > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:218)
> > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
> > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:758)
> > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> > Caused by: java.lang.RuntimeException: Unable to instantiate
> > org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> > at
> >
> >
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1139)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
> > at
> >
> >
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2284)
> > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2295)
> > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1136)
> > ... 18 more
> > Caused by: java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> > at
> >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1137)
> > ... 23 more
> > Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception
> > caught.
> > NestedThrowables:
> > java.lang.reflect.InvocationTargetException
> > at
> >
> >
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
> > at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
> > at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
> > at
> > org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:262)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:291)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:224)
> > at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:199)
> > at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
> > at
> >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:62)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:413)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:401)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:439)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:325)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:285)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:53)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:58)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4102)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
> > ... 28 more
> > Caused by: java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
> > at
> >
> >
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
> > ... 47 more
> > Caused by: java.lang.NoSuchFieldError: NUCLEUS_CONTEXT_LOADER
> > at org.datanucleus.NucleusContext.<clinit>(NucleusContext.java:105)
> > at
> >
> >
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:395)
> > at
> >
> >
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:294)
> > at
> >
> >
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
> > ... 55 more
> >
> > FAILED: Execution Error, return code 1 from
> > org.apache.hadoop.hive.ql.exec.DDLTask
> > 13/08/06 12:31:01 ERROR ql.Driver: FAILED: Execution Error, return code 1
> > from org.apache.hadoop.hive.ql.exec.DDLTask
> >
>
Re: DataNucleus patches for Hive
Posted by Eric Chu <ec...@rocketfuel.com>.
Xuefu,
Indeed that's the problem and "ant very-clean" also fixed the problem.
Thanks,
Eric
On Tue, Aug 6, 2013 at 5:10 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
> You probably have mix-matched datanucleus libraries in your lib/ folder.
> You should only three 3.2.x files if you do "ls datanu*". To avoid that,
> you need to do "ant very-clean" before building hive.
>
> --Xuefu
>
>
> On Tue, Aug 6, 2013 at 4:39 PM, Eric Chu <ec...@rocketfuel.com> wrote:
>
> > Hi,
> >
> > I'm a bit confused about what DataNucleus patches we should get for *Hive
> > 0.11 with JDK 6*. It'd be great if people working on that could shed some
> > light on the subject. Thanks in advance!
> >
> > After installing Hive 0.11 and applying the patch for HIVE-4619 (or else
> MR
> > queries will result in error), I will sometimes (but not always) see
> > DataNucleus error log messages as below:
> >
> > 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
> > requires "org.eclipse.core.resources" but it cannot be resolved.
> > 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
> > requires "org.eclipse.core.runtime" but it cannot be resolved.
> > 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
> > requires "org.eclipse.text" but it cannot be resolved.
> >
> > HIVE-2015 was opened for this issue, but the last comment there said it
> was
> > fixed through HIVE-3632, whose purpose is to upgrade DataNucleus to
> support
> > JDK 7. HIVE-3632 in turn was blocked by HIVE-4900 and broke HIVE-4942,
> both
> > of which have patches available in 12. So I ended up applying HIVE-4900,
> > HIVE-3632, and HIVE-4942, in that order. However, afterward, any query
> (MR,
> > select *, show tables) would result in error.
> >
> > In the end, after I applied the patch for HIVE-4900, I haven't seen those
> > error messages and queries are still running fine. Are the other patches
> > not compatible with JDK 6?
> >
> > *Error messages for SHOW TABLES after applying HIVE-4900, HIVE-3632, and
> > HIVE-4942:*
> >
> > FAILED: Error in metadata: java.lang.RuntimeException: Unable to
> > instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> > 13/08/06 12:31:01 ERROR exec.Task: FAILED: Error in metadata:
> > java.lang.RuntimeException: Unable to instantiate
> > org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> > org.apache.hadoop.hive.ql.metadata.HiveException:
> > java.lang.RuntimeException: Unable to instantiate
> > org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1140)
> > at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1125)
> > at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2265)
> > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:337)
> > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)
> > at
> >
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1361)
> > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1146)
> > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:948)
> > at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:261)
> > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:218)
> > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
> > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:758)
> > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> > Caused by: java.lang.RuntimeException: Unable to instantiate
> > org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> > at
> >
> >
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1139)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
> > at
> >
> >
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2284)
> > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2295)
> > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1136)
> > ... 18 more
> > Caused by: java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> > at
> >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1137)
> > ... 23 more
> > Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception
> > caught.
> > NestedThrowables:
> > java.lang.reflect.InvocationTargetException
> > at
> >
> >
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
> > at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
> > at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
> > at
> > org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:262)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:291)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:224)
> > at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:199)
> > at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
> > at
> >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:62)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:413)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:401)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:439)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:325)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:285)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:53)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:58)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4102)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
> > ... 28 more
> > Caused by: java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
> > at
> >
> >
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
> > ... 47 more
> > Caused by: java.lang.NoSuchFieldError: NUCLEUS_CONTEXT_LOADER
> > at org.datanucleus.NucleusContext.<clinit>(NucleusContext.java:105)
> > at
> >
> >
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:395)
> > at
> >
> >
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:294)
> > at
> >
> >
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
> > ... 55 more
> >
> > FAILED: Execution Error, return code 1 from
> > org.apache.hadoop.hive.ql.exec.DDLTask
> > 13/08/06 12:31:01 ERROR ql.Driver: FAILED: Execution Error, return code 1
> > from org.apache.hadoop.hive.ql.exec.DDLTask
> >
>
Re: DataNucleus patches for Hive
Posted by Xuefu Zhang <xz...@cloudera.com>.
You probably have mix-matched datanucleus libraries in your lib/ folder.
You should only three 3.2.x files if you do "ls datanu*". To avoid that,
you need to do "ant very-clean" before building hive.
--Xuefu
On Tue, Aug 6, 2013 at 4:39 PM, Eric Chu <ec...@rocketfuel.com> wrote:
> Hi,
>
> I'm a bit confused about what DataNucleus patches we should get for *Hive
> 0.11 with JDK 6*. It'd be great if people working on that could shed some
> light on the subject. Thanks in advance!
>
> After installing Hive 0.11 and applying the patch for HIVE-4619 (or else MR
> queries will result in error), I will sometimes (but not always) see
> DataNucleus error log messages as below:
>
> 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
> requires "org.eclipse.core.resources" but it cannot be resolved.
> 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
> requires "org.eclipse.core.runtime" but it cannot be resolved.
> 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core"
> requires "org.eclipse.text" but it cannot be resolved.
>
> HIVE-2015 was opened for this issue, but the last comment there said it was
> fixed through HIVE-3632, whose purpose is to upgrade DataNucleus to support
> JDK 7. HIVE-3632 in turn was blocked by HIVE-4900 and broke HIVE-4942, both
> of which have patches available in 12. So I ended up applying HIVE-4900,
> HIVE-3632, and HIVE-4942, in that order. However, afterward, any query (MR,
> select *, show tables) would result in error.
>
> In the end, after I applied the patch for HIVE-4900, I haven't seen those
> error messages and queries are still running fine. Are the other patches
> not compatible with JDK 6?
>
> *Error messages for SHOW TABLES after applying HIVE-4900, HIVE-3632, and
> HIVE-4942:*
>
> FAILED: Error in metadata: java.lang.RuntimeException: Unable to
> instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> 13/08/06 12:31:01 ERROR exec.Task: FAILED: Error in metadata:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1140)
> at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1125)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2265)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:337)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1361)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1146)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:948)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:261)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:218)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:758)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
>
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1139)
> at
>
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
> at
>
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
> at
>
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2284)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2295)
> at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1136)
> ... 18 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1137)
> ... 23 more
> Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception
> caught.
> NestedThrowables:
> java.lang.reflect.InvocationTargetException
> at
>
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:262)
> at
>
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:291)
> at
>
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:224)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:199)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
> at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
> at
>
> org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:62)
> at
>
> org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
> at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:413)
> at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:401)
> at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:439)
> at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:325)
> at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:285)
> at
>
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:53)
> at
>
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:58)
> at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4102)
> at
>
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
> ... 28 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
> at
>
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
> ... 47 more
> Caused by: java.lang.NoSuchFieldError: NUCLEUS_CONTEXT_LOADER
> at org.datanucleus.NucleusContext.<clinit>(NucleusContext.java:105)
> at
>
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:395)
> at
>
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:294)
> at
>
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
> ... 55 more
>
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> 13/08/06 12:31:01 ERROR ql.Driver: FAILED: Execution Error, return code 1
> from org.apache.hadoop.hive.ql.exec.DDLTask
>