You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Ehsan Haq <eh...@klarna.com> on 2013/01/23 13:41:50 UTC
serde jar causing problems in loading other jars.
Hi,
I have writen a custom serde and bundle it in a jar file, which is
working fine, when I add the jar using the CLI command add jar. However
when I put the jar in the hive/lib folder so that I dont have to explicitly
add the jar, it looks like other jars were failed to load. The outcome is
that the meta data is also not accessible due to that. The jar works just
fine if put it somewhere else and add it via add jar.
Any idea what might be wrong?
/Ehsan
Re: serde jar causing problems in loading other jars.
Posted by Ehsan Haq <eh...@klarna.com>.
Thanks Dean, i will try to remove unnesessary jars and come back with the
outcome.
On Jan 23, 2013 7:11 PM, "Dean Wampler" <de...@thinkbiganalytics.com>
wrote:
> I see from your listing that your jar contains the contents of many, if
> not all, of the Apache and logging jars that are also in the hadoop/lib and
> hive/lib directories, including the core hadoop and hive jars themselves.
> plus some Google, JSON, and other libraries. Most likely, this is causing
> issues, besides creating an unnecessarily large jar file of your code.
>
> You should remove ALL the contents of other jars from your jar and use ADD
> JAR only on the unique jars, like the Joda time jar you're using. I suspect
> that will reduce or eliminate the problems, even though that it can be
> tedious. I suspect you only really need to add a few extra jars, though.
>
> The root-cause exception :
>
> Caused by: java.lang.NullPointerException
> at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(
> NonManagedPluginRegistry.java:443)
> at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(
> NonManagedPluginRegistry.java:355)
> ...
>
> involves the $HIVE_HOME/lib/datanucleus-core-X.Y.Z.jar. I can only guess
> that loading multiple copies of other jar contents is tripping it up
> somehow.
>
> Good luck,
> dean
>
> On Wed, Jan 23, 2013 at 11:44 AM, Ehsan Haq <eh...@klarna.com> wrote:
>
>> I tried to rename my serde jar to zzzz.jar so that it would be loaded in
>> the last, but still the same behaviour. when i run "show tables;" on the
>> terminal, I get this exception in the logs. Also the list of the classes in
>> the serde jar files is added in the attachment
>>
>> 2013-01-23 18:29:59,852 ERROR exec.Task
>> (SessionState.java:printError(380)) - FAILED: Error in metadata:
>> javax.jdo.JDOFatalInternalException: Unexpected exception caught.
>> NestedThrowables:
>> java.lang.reflect.InvocationTargetException
>> org.apache.hadoop.hive.ql.metadata.HiveException:
>> javax.jdo.JDOFatalInternalException: Unexpected exception caught.
>> NestedThrowables:
>> java.lang.reflect.InvocationTargetException
>> at
>> org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1099)
>> at
>> org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1084)
>> at
>> org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:1957)
>> at
>> org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:601)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception
>> caught.
>> NestedThrowables:
>> java.lang.reflect.InvocationTargetException
>> at
>> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
>> at
>> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
>> at
>> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
>> at
>> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:246)
>> at
>> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:275)
>> at
>> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:208)
>> at
>> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:183)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:407)
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:359)
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:504)
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:266)
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:228)
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:114)
>> at
>> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2110)
>> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2120)
>> at
>> org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1095)
>> ... 18 more
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:601)
>> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
>> at
>> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
>> ... 35 more
>> Caused by: java.lang.NullPointerException
>> at
>> org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:443)
>> at
>> org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:355)
>> at
>> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:215)
>> at
>> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:156)
>> at
>> org.datanucleus.plugin.PluginManager.registerExtensionPoints(PluginManager.java:82)
>> at org.datanucleus.OMFContext.<init>(OMFContext.java:156)
>> at org.datanucleus.OMFContext.<init>(OMFContext.java:137)
>> at
>> org.datanucleus.ObjectManagerFactoryImpl.initialiseOMFContext(ObjectManagerFactoryImpl.java:132)
>> at
>> org.datanucleus.jdo.JDOPersistenceManagerFactory.initialiseProperties(JDOPersistenceManagerFactory.java:363)
>> at
>> org.datanucleus.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:307)
>> at
>> org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:255)
>> at
>> org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182)
>> ... 43 more
>>
>>
>>
>> On Wed, Jan 23, 2013 at 3:05 PM, Ehsan Haq <eh...@klarna.com> wrote:
>>
>>> Thanks dean, I knew about the .hiverc script, will try the other
>>> alternative of renaming jar and get back. Thanks
>>>
>>> /Ehsan
>>>
>>>
>>> On Wed, Jan 23, 2013 at 2:57 PM, Dean Wampler <
>>> dean.wampler@thinkbiganalytics.com> wrote:
>>>
>>>> Is there anything in the logs about problems loading the jar, etc.?
>>>>
>>>> The jar files in $HVE_HOME are added to the CLASSPATH in alphabetical
>>>> order. As an experiment, rename your jar with a name that will go last,
>>>> something like zzz.jar, and see what happens when you start Hive. If it
>>>> seems to be working normally, it may be that some file in your jar file has
>>>> a name that's the same as a file in one of Hive's jar files and your's gets
>>>> read instead. An XML config file, for example.
>>>>
>>>> If this experiment appears to work, run "jar tf name-of-your-file.jar"
>>>> and post the listing here, if you don't mind.
>>>>
>>>> Note that an alternative to dropping jar files in Hive's lib directory
>>>> is to put the "add jar ..." command in your $HOME/.hiverc file, which Hive
>>>> will read on startup (Hive v0.7.0 and later). What Hive version are you
>>>> using, by the way? I also put lots of custom configuration setting
>>>> commands, e.g., set hive.exec.mode.local.auto=true; (to encourage local
>>>> mode execution, when possible).
>>>>
>>>> dean
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 6:41 AM, Ehsan Haq <eh...@klarna.com>wrote:
>>>>
>>>>> Hi,
>>>>> I have writen a custom serde and bundle it in a jar file, which is
>>>>> working fine, when I add the jar using the CLI command add jar. However
>>>>> when I put the jar in the hive/lib folder so that I dont have to explicitly
>>>>> add the jar, it looks like other jars were failed to load. The outcome is
>>>>> that the meta data is also not accessible due to that. The jar works just
>>>>> fine if put it somewhere else and add it via add jar.
>>>>> Any idea what might be wrong?
>>>>>
>>>>> /Ehsan
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Dean Wampler, Ph.D.*
>>>> thinkbiganalytics.com
>>>> +1-312-339-1330
>>>>
>>>>
>>>
>>>
>>> --
>>> *Muhammad Ehsan ul Haque*
>>> Klarna AB
>>> Norra Stationsgatan 61
>>> SE-113 43 Stockholm
>>>
>>> Tel: +46 (0)8- 120 120 00
>>> Fax: +46 (0)8- 120 120 99
>>> Web: www.klarna.com
>>>
>>
>>
>>
>> --
>> *Muhammad Ehsan ul Haque*
>> Klarna AB
>> Norra Stationsgatan 61
>> SE-113 43 Stockholm
>>
>> Tel: +46 (0)8- 120 120 00
>> Fax: +46 (0)8- 120 120 99
>> Web: www.klarna.com
>>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
Re: serde jar causing problems in loading other jars.
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
I see from your listing that your jar contains the contents of many, if not
all, of the Apache and logging jars that are also in the hadoop/lib and
hive/lib directories, including the core hadoop and hive jars themselves.
plus some Google, JSON, and other libraries. Most likely, this is causing
issues, besides creating an unnecessarily large jar file of your code.
You should remove ALL the contents of other jars from your jar and use ADD
JAR only on the unique jars, like the Joda time jar you're using. I suspect
that will reduce or eliminate the problems, even though that it can be
tedious. I suspect you only really need to add a few extra jars, though.
The root-cause exception :
Caused by: java.lang.NullPointerException
at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(
NonManagedPluginRegistry.java:443)
at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(
NonManagedPluginRegistry.java:355)
...
involves the $HIVE_HOME/lib/datanucleus-core-X.Y.Z.jar. I can only guess
that loading multiple copies of other jar contents is tripping it up
somehow.
Good luck,
dean
On Wed, Jan 23, 2013 at 11:44 AM, Ehsan Haq <eh...@klarna.com> wrote:
> I tried to rename my serde jar to zzzz.jar so that it would be loaded in
> the last, but still the same behaviour. when i run "show tables;" on the
> terminal, I get this exception in the logs. Also the list of the classes in
> the serde jar files is added in the attachment
>
> 2013-01-23 18:29:59,852 ERROR exec.Task
> (SessionState.java:printError(380)) - FAILED: Error in metadata:
> javax.jdo.JDOFatalInternalException: Unexpected exception caught.
> NestedThrowables:
> java.lang.reflect.InvocationTargetException
> org.apache.hadoop.hive.ql.metadata.HiveException:
> javax.jdo.JDOFatalInternalException: Unexpected exception caught.
> NestedThrowables:
> java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1099)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1084)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:1957)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception
> caught.
> NestedThrowables:
> java.lang.reflect.InvocationTargetException
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
> at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
> at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:246)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:275)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:208)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:183)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:407)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:359)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:504)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:266)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:228)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:114)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2110)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2120)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1095)
> ... 18 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
> ... 35 more
> Caused by: java.lang.NullPointerException
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:443)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:355)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:215)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:156)
> at
> org.datanucleus.plugin.PluginManager.registerExtensionPoints(PluginManager.java:82)
> at org.datanucleus.OMFContext.<init>(OMFContext.java:156)
> at org.datanucleus.OMFContext.<init>(OMFContext.java:137)
> at
> org.datanucleus.ObjectManagerFactoryImpl.initialiseOMFContext(ObjectManagerFactoryImpl.java:132)
> at
> org.datanucleus.jdo.JDOPersistenceManagerFactory.initialiseProperties(JDOPersistenceManagerFactory.java:363)
> at
> org.datanucleus.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:307)
> at
> org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:255)
> at
> org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182)
> ... 43 more
>
>
>
> On Wed, Jan 23, 2013 at 3:05 PM, Ehsan Haq <eh...@klarna.com> wrote:
>
>> Thanks dean, I knew about the .hiverc script, will try the other
>> alternative of renaming jar and get back. Thanks
>>
>> /Ehsan
>>
>>
>> On Wed, Jan 23, 2013 at 2:57 PM, Dean Wampler <
>> dean.wampler@thinkbiganalytics.com> wrote:
>>
>>> Is there anything in the logs about problems loading the jar, etc.?
>>>
>>> The jar files in $HVE_HOME are added to the CLASSPATH in alphabetical
>>> order. As an experiment, rename your jar with a name that will go last,
>>> something like zzz.jar, and see what happens when you start Hive. If it
>>> seems to be working normally, it may be that some file in your jar file has
>>> a name that's the same as a file in one of Hive's jar files and your's gets
>>> read instead. An XML config file, for example.
>>>
>>> If this experiment appears to work, run "jar tf name-of-your-file.jar"
>>> and post the listing here, if you don't mind.
>>>
>>> Note that an alternative to dropping jar files in Hive's lib directory
>>> is to put the "add jar ..." command in your $HOME/.hiverc file, which Hive
>>> will read on startup (Hive v0.7.0 and later). What Hive version are you
>>> using, by the way? I also put lots of custom configuration setting
>>> commands, e.g., set hive.exec.mode.local.auto=true; (to encourage local
>>> mode execution, when possible).
>>>
>>> dean
>>>
>>>
>>> On Wed, Jan 23, 2013 at 6:41 AM, Ehsan Haq <eh...@klarna.com> wrote:
>>>
>>>> Hi,
>>>> I have writen a custom serde and bundle it in a jar file, which is
>>>> working fine, when I add the jar using the CLI command add jar. However
>>>> when I put the jar in the hive/lib folder so that I dont have to explicitly
>>>> add the jar, it looks like other jars were failed to load. The outcome is
>>>> that the meta data is also not accessible due to that. The jar works just
>>>> fine if put it somewhere else and add it via add jar.
>>>> Any idea what might be wrong?
>>>>
>>>> /Ehsan
>>>>
>>>
>>>
>>>
>>> --
>>> *Dean Wampler, Ph.D.*
>>> thinkbiganalytics.com
>>> +1-312-339-1330
>>>
>>>
>>
>>
>> --
>> *Muhammad Ehsan ul Haque*
>> Klarna AB
>> Norra Stationsgatan 61
>> SE-113 43 Stockholm
>>
>> Tel: +46 (0)8- 120 120 00
>> Fax: +46 (0)8- 120 120 99
>> Web: www.klarna.com
>>
>
>
>
> --
> *Muhammad Ehsan ul Haque*
> Klarna AB
> Norra Stationsgatan 61
> SE-113 43 Stockholm
>
> Tel: +46 (0)8- 120 120 00
> Fax: +46 (0)8- 120 120 99
> Web: www.klarna.com
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: serde jar causing problems in loading other jars.
Posted by Ehsan Haq <eh...@klarna.com>.
I tried to rename my serde jar to zzzz.jar so that it would be loaded in
the last, but still the same behaviour. when i run "show tables;" on the
terminal, I get this exception in the logs. Also the list of the classes in
the serde jar files is added in the attachment
2013-01-23 18:29:59,852 ERROR exec.Task (SessionState.java:printError(380))
- FAILED: Error in metadata: javax.jdo.JDOFatalInternalException:
Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
org.apache.hadoop.hive.ql.metadata.HiveException:
javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1099)
at
org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1084)
at
org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:1957)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
at
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
at
javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
at
javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:246)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:275)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:208)
at
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:183)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:407)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:359)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:504)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:266)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:228)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:114)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2110)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2120)
at
org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1095)
... 18 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
at
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
... 35 more
Caused by: java.lang.NullPointerException
at
org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:443)
at
org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:355)
at
org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:215)
at
org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:156)
at
org.datanucleus.plugin.PluginManager.registerExtensionPoints(PluginManager.java:82)
at org.datanucleus.OMFContext.<init>(OMFContext.java:156)
at org.datanucleus.OMFContext.<init>(OMFContext.java:137)
at
org.datanucleus.ObjectManagerFactoryImpl.initialiseOMFContext(ObjectManagerFactoryImpl.java:132)
at
org.datanucleus.jdo.JDOPersistenceManagerFactory.initialiseProperties(JDOPersistenceManagerFactory.java:363)
at
org.datanucleus.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:307)
at
org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:255)
at
org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182)
... 43 more
On Wed, Jan 23, 2013 at 3:05 PM, Ehsan Haq <eh...@klarna.com> wrote:
> Thanks dean, I knew about the .hiverc script, will try the other
> alternative of renaming jar and get back. Thanks
>
> /Ehsan
>
>
> On Wed, Jan 23, 2013 at 2:57 PM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:
>
>> Is there anything in the logs about problems loading the jar, etc.?
>>
>> The jar files in $HVE_HOME are added to the CLASSPATH in alphabetical
>> order. As an experiment, rename your jar with a name that will go last,
>> something like zzz.jar, and see what happens when you start Hive. If it
>> seems to be working normally, it may be that some file in your jar file has
>> a name that's the same as a file in one of Hive's jar files and your's gets
>> read instead. An XML config file, for example.
>>
>> If this experiment appears to work, run "jar tf name-of-your-file.jar"
>> and post the listing here, if you don't mind.
>>
>> Note that an alternative to dropping jar files in Hive's lib directory is
>> to put the "add jar ..." command in your $HOME/.hiverc file, which Hive
>> will read on startup (Hive v0.7.0 and later). What Hive version are you
>> using, by the way? I also put lots of custom configuration setting
>> commands, e.g., set hive.exec.mode.local.auto=true; (to encourage local
>> mode execution, when possible).
>>
>> dean
>>
>>
>> On Wed, Jan 23, 2013 at 6:41 AM, Ehsan Haq <eh...@klarna.com> wrote:
>>
>>> Hi,
>>> I have writen a custom serde and bundle it in a jar file, which is
>>> working fine, when I add the jar using the CLI command add jar. However
>>> when I put the jar in the hive/lib folder so that I dont have to explicitly
>>> add the jar, it looks like other jars were failed to load. The outcome is
>>> that the meta data is also not accessible due to that. The jar works just
>>> fine if put it somewhere else and add it via add jar.
>>> Any idea what might be wrong?
>>>
>>> /Ehsan
>>>
>>
>>
>>
>> --
>> *Dean Wampler, Ph.D.*
>> thinkbiganalytics.com
>> +1-312-339-1330
>>
>>
>
>
> --
> *Muhammad Ehsan ul Haque*
> Klarna AB
> Norra Stationsgatan 61
> SE-113 43 Stockholm
>
> Tel: +46 (0)8- 120 120 00
> Fax: +46 (0)8- 120 120 99
> Web: www.klarna.com
>
--
*Muhammad Ehsan ul Haque*
Klarna AB
Norra Stationsgatan 61
SE-113 43 Stockholm
Tel: +46 (0)8- 120 120 00
Fax: +46 (0)8- 120 120 99
Web: www.klarna.com
Re: serde jar causing problems in loading other jars.
Posted by Ehsan Haq <eh...@klarna.com>.
Thanks dean, I knew about the .hiverc script, will try the other
alternative of renaming jar and get back. Thanks
/Ehsan
On Wed, Jan 23, 2013 at 2:57 PM, Dean Wampler <
dean.wampler@thinkbiganalytics.com> wrote:
> Is there anything in the logs about problems loading the jar, etc.?
>
> The jar files in $HVE_HOME are added to the CLASSPATH in alphabetical
> order. As an experiment, rename your jar with a name that will go last,
> something like zzz.jar, and see what happens when you start Hive. If it
> seems to be working normally, it may be that some file in your jar file has
> a name that's the same as a file in one of Hive's jar files and your's gets
> read instead. An XML config file, for example.
>
> If this experiment appears to work, run "jar tf name-of-your-file.jar" and
> post the listing here, if you don't mind.
>
> Note that an alternative to dropping jar files in Hive's lib directory is
> to put the "add jar ..." command in your $HOME/.hiverc file, which Hive
> will read on startup (Hive v0.7.0 and later). What Hive version are you
> using, by the way? I also put lots of custom configuration setting
> commands, e.g., set hive.exec.mode.local.auto=true; (to encourage local
> mode execution, when possible).
>
> dean
>
>
> On Wed, Jan 23, 2013 at 6:41 AM, Ehsan Haq <eh...@klarna.com> wrote:
>
>> Hi,
>> I have writen a custom serde and bundle it in a jar file, which is
>> working fine, when I add the jar using the CLI command add jar. However
>> when I put the jar in the hive/lib folder so that I dont have to explicitly
>> add the jar, it looks like other jars were failed to load. The outcome is
>> that the meta data is also not accessible due to that. The jar works just
>> fine if put it somewhere else and add it via add jar.
>> Any idea what might be wrong?
>>
>> /Ehsan
>>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
--
*Muhammad Ehsan ul Haque*
Klarna AB
Norra Stationsgatan 61
SE-113 43 Stockholm
Tel: +46 (0)8- 120 120 00
Fax: +46 (0)8- 120 120 99
Web: www.klarna.com
Re: serde jar causing problems in loading other jars.
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
Is there anything in the logs about problems loading the jar, etc.?
The jar files in $HVE_HOME are added to the CLASSPATH in alphabetical
order. As an experiment, rename your jar with a name that will go last,
something like zzz.jar, and see what happens when you start Hive. If it
seems to be working normally, it may be that some file in your jar file has
a name that's the same as a file in one of Hive's jar files and your's gets
read instead. An XML config file, for example.
If this experiment appears to work, run "jar tf name-of-your-file.jar" and
post the listing here, if you don't mind.
Note that an alternative to dropping jar files in Hive's lib directory is
to put the "add jar ..." command in your $HOME/.hiverc file, which Hive
will read on startup (Hive v0.7.0 and later). What Hive version are you
using, by the way? I also put lots of custom configuration setting
commands, e.g., set hive.exec.mode.local.auto=true; (to encourage local
mode execution, when possible).
dean
On Wed, Jan 23, 2013 at 6:41 AM, Ehsan Haq <eh...@klarna.com> wrote:
> Hi,
> I have writen a custom serde and bundle it in a jar file, which is
> working fine, when I add the jar using the CLI command add jar. However
> when I put the jar in the hive/lib folder so that I dont have to explicitly
> add the jar, it looks like other jars were failed to load. The outcome is
> that the meta data is also not accessible due to that. The jar works just
> fine if put it somewhere else and add it via add jar.
> Any idea what might be wrong?
>
> /Ehsan
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330