You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Josh Ferguson <jo...@besquared.net> on 2009/01/24 21:48:27 UTC
Still can't get metastore working with mysql
Ok I still can't manage to get the metastore working with mysql after
all this time trying. The error is:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
The relevant config is:
<!-- Hadoop Setup -->
<property>
<name>hadoop.bin.path</name>
<value>/usr/local/hadoop/bin/hadoop</value>
<description>Path to hadoop binary. Assumes that by default we are
executing from hive</description>
</property>
<property>
<name>hadoop.config.dir</name>
<value>/usr/local/hadoop/conf</value>
<description>Path to hadoop configuration. Again assumes that by
default we are executing from hive/</description>
</property>
<!-- Hive Execution Parameters -->
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive-${user.name}</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>******</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>******</value>
</property>
<property>
<name>org.jpox.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>org.jpox.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.usefilestore</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.checkForDefaultDb</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.rawstore.impl</name>
<value>org.apache.hadoop.hive.metastore.ObjectStore</value>
<description>Name of the class that implements
org.apache.hadoop.hive.metastore.rawstore interface. This class is
used to store and retrieval of raw metadata objects such as table,
database</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server
or open a new metastore server in Hive Client JVM</description>
</property>
Re: Still can't get metastore working with mysql
Posted by Prasad Chakka <pr...@facebook.com>.
If you had just checked the wiki here http://wiki.apache.org/hadoop/Hive/AdminManual/MetastoreAdmin#head-baa6450ac33f1324e55dcb0d52344a440f14d75e
But was only created last week :)
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 15:08:08 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
That's why I had tried to do initially but the errors I was seeing made me think my configuration was incorrect so I was changing the configuration stuff. This all should be solved in the future with some wiki love. My guess is most people are going to want to use mysql as their metadata store and not derby. Derby is fine developing on your local machine but not much else. Hadoop has a good write up on their wiki about local, psuedo-distributed, and fully distributed setups. Hive could probably benefit from something similar as making the shift from just prototyping on your local machine to a production deployment can be a difficult step.
Josh F.
On Jan 24, 2009, at 3:00 PM, Prasad Chakka wrote:
Well, if you had just used the defaults that are in the hive source and changed just the JDBC params, that is what will happen. The parameters that you are using comes from internal Facebook settings and they are tuned for production. Checking for correctness of schema every time hive connects to metastore is inefficient in production cluster.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 14:55:21 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
Joydeep Sen Sarma sent them out in the list last month when I was first trying to get this working. I tried to modify them appropriately.
This whole "lazy" creation of schemas and tables seems like a really bad way to do things. My configuration shouldn't be different for something that is up and running and something that needs to be created. I should just be able to define my connection parameters and then have a command that migrates my database into the proper state for use. Trying to figure out what magical set of esoteric configuration parameters will tell this black box library to create my schema is really frustrating.
Josh F.
On Jan 24, 2009, at 2:26 PM, Prasad Chakka wrote:
Try setting fixedDataStore to false.
BTB, where did you get the initial values from the config you posted? These values are for the store that is already running in production and you don't expect any changes to the schema.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 14:14:14 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
That got me a little further, now I'm getting this error:
FAILED: Error in metadata: MetaException(message:Got exception: javax.jdo.JDODataStoreException Required table missing : "`SERDES`" in Catalog "" Schema "". JPOX requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "org.jpox.autoCreateTables")
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
In my hive-default.xml I turned on autoCreateTables and autoCreateSchema and it's still throwing this error back to me? Is there something else I need to do?
Josh Ferguson
On Jan 24, 2009, at 1:28 PM, Prasad Chakka wrote:
Down load the jar from here http://dev.mysql.com/downloads/connector/j/5.1.html
________________________________
From: Prasad Chakka <pr...@facebook.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:13:35 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
JPOX/Hive doesn't yet download the relevant client libraries based on the class name :) Drop mysql client jar in hive/lib. Also try setting org.jpox.autoCreateSchema to true.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:06:26 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
So I'm not really sure what the state of the database should be, I have a database called hive and the user has permissions to access it. I'm not sure which mysql client libraries need to be in the classpath, I assumed it was all rolled into the jpox stuff. Here is the complete error:
09/01/24 16:02:16 DEBUG exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:446)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:433)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:330)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:444)
... 16 more
On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
There could be many reasons for it. Could you post the stacktrace for the JDOFatal?
Some reasons could be
That mysql client libs are in the classpath.
That there is a database called 'hive' in mysql
That there is schema already created in that database (otherwise set autoCreateSchema to true) etc...
Check documentation for the many jpox options
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 12:48:27 -0800
To: <hi...@hadoop.apache.org>
Subject: Still can't get metastore working with mysql
Ok I still can't manage to get the metastore working with mysql after
all this time trying. The error is:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
The relevant config is:
<!-- Hadoop Setup -->
<property>
<name>hadoop.bin.path</name>
<value>/usr/local/hadoop/bin/hadoop</value>
<description>Path to hadoop binary. Assumes that by default we are
executing from hive</description>
</property>
<property>
<name>hadoop.config.dir</name>
<value>/usr/local/hadoop/conf</value>
<description>Path to hadoop configuration. Again assumes that by
default we are executing from hive/</description>
</property>
<!-- Hive Execution Parameters -->
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive-${user.name}</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>******</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>******</value>
</property>
<property>
<name>org.jpox.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>org.jpox.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.usefilestore</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.checkForDefaultDb</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.rawstore.impl</name>
<value>org.apache.hadoop.hive.metastore.ObjectStore</value>
<description>Name of the class that implements
org.apache.hadoop.hive.metastore.rawstore interface. This class is
used to store and retrieval of raw metadata objects such as table,
database</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server
or open a new metastore server in Hive Client JVM</description>
</property>
Re: Still can't get metastore working with mysql
Posted by Josh Ferguson <jo...@besquared.net>.
That's why I had tried to do initially but the errors I was seeing
made me think my configuration was incorrect so I was changing the
configuration stuff. This all should be solved in the future with some
wiki love. My guess is most people are going to want to use mysql as
their metadata store and not derby. Derby is fine developing on your
local machine but not much else. Hadoop has a good write up on their
wiki about local, psuedo-distributed, and fully distributed setups.
Hive could probably benefit from something similar as making the shift
from just prototyping on your local machine to a production deployment
can be a difficult step.
Josh F.
On Jan 24, 2009, at 3:00 PM, Prasad Chakka wrote:
> Well, if you had just used the defaults that are in the hive source
> and changed just the JDBC params, that is what will happen. The
> parameters that you are using comes from internal Facebook settings
> and they are tuned for production. Checking for correctness of
> schema every time hive connects to metastore is inefficient in
> production cluster.
>
>
> From: Josh Ferguson <jo...@besquared.net>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 14:55:21 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Re: Still can't get metastore working with mysql
>
> Joydeep Sen Sarma sent them out in the list last month when I was
> first trying to get this working. I tried to modify them
> appropriately.
>
> This whole "lazy" creation of schemas and tables seems like a really
> bad way to do things. My configuration shouldn't be different for
> something that is up and running and something that needs to be
> created. I should just be able to define my connection parameters
> and then have a command that migrates my database into the proper
> state for use. Trying to figure out what magical set of esoteric
> configuration parameters will tell this black box library to create
> my schema is really frustrating.
>
> Josh F.
>
> On Jan 24, 2009, at 2:26 PM, Prasad Chakka wrote:
>
> Try setting fixedDataStore to false.
>
> BTB, where did you get the initial values from the config you
> posted? These values are for the store that is already running in
> production and you don’t expect any changes to the schema.
>
>
>
> From: Josh Ferguson <jo...@besquared.net>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 14:14:14 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Re: Still can't get metastore working with mysql
>
> That got me a little further, now I'm getting this error:
>
> FAILED: Error in metadata: MetaException(message:Got exception:
> javax.jdo.JDODataStoreException Required table missing : "`SERDES`"
> in Catalog "" Schema "". JPOX requires this table to perform its
> persistence operations. Either your MetaData is incorrect, or you
> need to enable "org.jpox.autoCreateTables")
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> In my hive-default.xml I turned on autoCreateTables and
> autoCreateSchema and it's still throwing this error back to me? Is
> there something else I need to do?
>
> Josh Ferguson
>
> On Jan 24, 2009, at 1:28 PM, Prasad Chakka wrote:
>
>
> Down load the jar from here http://dev.mysql.com/downloads/connector/j/5.1.html
>
>
>
>
> From: Prasad Chakka <pr...@facebook.com>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 13:13:35 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Re: Still can't get metastore working with mysql
>
> JPOX/Hive doesn’t yet download the relevant client libraries based
> on the class name :) Drop mysql client jar in hive/lib. Also try
> setting org.jpox.autoCreateSchema to true.
>
>
>
> From: Josh Ferguson <jo...@besquared.net>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 13:06:26 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Re: Still can't get metastore working with mysql
>
> So I'm not really sure what the state of the database should be, I
> have a database called hive and the user has permissions to access
> it. I'm not sure which mysql client libraries need to be in the
> classpath, I assumed it was all rolled into the jpox stuff. Here is
> the complete error:
>
> 09/01/24 16:02:16 DEBUG exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got
> exception: javax.jdo.JDOFatalInternalException Error creating
> transactional connection factory)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
> 446)
> at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:
> 433)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:
> 330)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:
> 174)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:
> 207)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun
> .reflect
> .NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun
> .reflect
> .DelegatingMethodAccessorImpl
> .invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
> Caused by: MetaException(message:Got exception:
> javax.jdo.JDOFatalInternalException Error creating transactional
> connection factory)
> at
> org
> .apache
> .hadoop
> .hive
> .metastore
> .MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
> at
> org
> .apache
> .hadoop
> .hive
> .metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
> 444)
> ... 16 more
>
>
> On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
>
>
>
> There could be many reasons for it. Could you post the stacktrace
> for the JDOFatal?
>
> Some reasons could be
> That mysql client libs are in the classpath.
> That there is a database called ‘hive’ in mysql
> That there is schema already created in that database (otherwise
> set autoCreateSchema to true) etc...
>
> Check documentation for the many jpox options
>
>
>
> From: Josh Ferguson <jo...@besquared.net>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 12:48:27 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Still can't get metastore working with mysql
>
> Ok I still can't manage to get the metastore working with mysql
> after
> all this time trying. The error is:
>
> FAILED: Error in metadata: MetaException(message:Got exception:
> javax.jdo.JDOFatalInternalException Error creating transactional
> connection factory)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> The relevant config is:
>
> <!-- Hadoop Setup -->
> <property>
> <name>hadoop.bin.path</name>
> <value>/usr/local/hadoop/bin/hadoop</value>
> <description>Path to hadoop binary. Assumes that by default we
> are
> executing from hive</description>
> </property>
>
> <property>
> <name>hadoop.config.dir</name>
> <value>/usr/local/hadoop/conf</value>
> <description>Path to hadoop configuration. Again assumes that by
> default we are executing from hive/</description>
> </property>
>
> <!-- Hive Execution Parameters -->
> <property>
> <name>hive.exec.scratchdir</name>
> <value>/tmp/hive-${user.name}</value>
> <description>Scratch space for Hive jobs</description>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://localhost/hive</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>******</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>******</value>
> </property>
>
> <property>
> <name>org.jpox.autoCreateSchema</name>
> <value>false</value>
> </property>
>
> <property>
> <name>org.jpox.fixedDatastore</name>
> <value>true</value>
> </property>
>
> <property>
> <name>hive.metastore.usefilestore</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.metastore.checkForDefaultDb</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.metastore.rawstore.impl</name>
> <value>org.apache.hadoop.hive.metastore.ObjectStore</value>
> <description>Name of the class that implements
> org.apache.hadoop.hive.metastore.rawstore interface. This class is
> used to store and retrieval of raw metadata objects such as table,
> database</description>
> </property>
>
> <property>
> <name>hive.metastore.local</name>
> <value>true</value>
> <description>controls whether to connect to remove metastore
> server
> or open a new metastore server in Hive Client JVM</description>
> </property>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Still can't get metastore working with mysql
Posted by Prasad Chakka <pr...@facebook.com>.
On the other hand, you can generate SQL schema using "generate-schem" ant target in hive/metastore and use the config parameters that Joydeep sent.
________________________________
From: Prasad Chakka <pr...@facebook.com>
Date: Sat, 24 Jan 2009 15:00:00 -0800
To: <hi...@hadoop.apache.org>
Conversation: Still can't get metastore working with mysql
Subject: Re: Still can't get metastore working with mysql
Well, if you had just used the defaults that are in the hive source and changed just the JDBC params, that is what will happen. The parameters that you are using comes from internal Facebook settings and they are tuned for production. Checking for correctness of schema every time hive connects to metastore is inefficient in production cluster.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 14:55:21 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
Joydeep Sen Sarma sent them out in the list last month when I was first trying to get this working. I tried to modify them appropriately.
This whole "lazy" creation of schemas and tables seems like a really bad way to do things. My configuration shouldn't be different for something that is up and running and something that needs to be created. I should just be able to define my connection parameters and then have a command that migrates my database into the proper state for use. Trying to figure out what magical set of esoteric configuration parameters will tell this black box library to create my schema is really frustrating.
Josh F.
On Jan 24, 2009, at 2:26 PM, Prasad Chakka wrote:
Try setting fixedDataStore to false.
BTB, where did you get the initial values from the config you posted? These values are for the store that is already running in production and you don't expect any changes to the schema.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 14:14:14 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
That got me a little further, now I'm getting this error:
FAILED: Error in metadata: MetaException(message:Got exception: javax.jdo.JDODataStoreException Required table missing : "`SERDES`" in Catalog "" Schema "". JPOX requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "org.jpox.autoCreateTables")
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
In my hive-default.xml I turned on autoCreateTables and autoCreateSchema and it's still throwing this error back to me? Is there something else I need to do?
Josh Ferguson
On Jan 24, 2009, at 1:28 PM, Prasad Chakka wrote:
Down load the jar from here http://dev.mysql.com/downloads/connector/j/5.1.html
________________________________
From: Prasad Chakka <pr...@facebook.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:13:35 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
JPOX/Hive doesn't yet download the relevant client libraries based on the class name :) Drop mysql client jar in hive/lib. Also try setting org.jpox.autoCreateSchema to true.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:06:26 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
So I'm not really sure what the state of the database should be, I have a database called hive and the user has permissions to access it. I'm not sure which mysql client libraries need to be in the classpath, I assumed it was all rolled into the jpox stuff. Here is the complete error:
09/01/24 16:02:16 DEBUG exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:446)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:433)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:330)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:444)
... 16 more
On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
There could be many reasons for it. Could you post the stacktrace for the JDOFatal?
Some reasons could be
That mysql client libs are in the classpath.
That there is a database called 'hive' in mysql
That there is schema already created in that database (otherwise set autoCreateSchema to true) etc...
Check documentation for the many jpox options
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 12:48:27 -0800
To: <hi...@hadoop.apache.org>
Subject: Still can't get metastore working with mysql
Ok I still can't manage to get the metastore working with mysql after
all this time trying. The error is:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
The relevant config is:
<!-- Hadoop Setup -->
<property>
<name>hadoop.bin.path</name>
<value>/usr/local/hadoop/bin/hadoop</value>
<description>Path to hadoop binary. Assumes that by default we are
executing from hive</description>
</property>
<property>
<name>hadoop.config.dir</name>
<value>/usr/local/hadoop/conf</value>
<description>Path to hadoop configuration. Again assumes that by
default we are executing from hive/</description>
</property>
<!-- Hive Execution Parameters -->
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive-${user.name}</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>******</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>******</value>
</property>
<property>
<name>org.jpox.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>org.jpox.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.usefilestore</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.checkForDefaultDb</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.rawstore.impl</name>
<value>org.apache.hadoop.hive.metastore.ObjectStore</value>
<description>Name of the class that implements
org.apache.hadoop.hive.metastore.rawstore interface. This class is
used to store and retrieval of raw metadata objects such as table,
database</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server
or open a new metastore server in Hive Client JVM</description>
</property>
Re: Still can't get metastore working with mysql
Posted by Prasad Chakka <pr...@facebook.com>.
Well, if you had just used the defaults that are in the hive source and changed just the JDBC params, that is what will happen. The parameters that you are using comes from internal Facebook settings and they are tuned for production. Checking for correctness of schema every time hive connects to metastore is inefficient in production cluster.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 14:55:21 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
Joydeep Sen Sarma sent them out in the list last month when I was first trying to get this working. I tried to modify them appropriately.
This whole "lazy" creation of schemas and tables seems like a really bad way to do things. My configuration shouldn't be different for something that is up and running and something that needs to be created. I should just be able to define my connection parameters and then have a command that migrates my database into the proper state for use. Trying to figure out what magical set of esoteric configuration parameters will tell this black box library to create my schema is really frustrating.
Josh F.
On Jan 24, 2009, at 2:26 PM, Prasad Chakka wrote:
Try setting fixedDataStore to false.
BTB, where did you get the initial values from the config you posted? These values are for the store that is already running in production and you don't expect any changes to the schema.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 14:14:14 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
That got me a little further, now I'm getting this error:
FAILED: Error in metadata: MetaException(message:Got exception: javax.jdo.JDODataStoreException Required table missing : "`SERDES`" in Catalog "" Schema "". JPOX requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "org.jpox.autoCreateTables")
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
In my hive-default.xml I turned on autoCreateTables and autoCreateSchema and it's still throwing this error back to me? Is there something else I need to do?
Josh Ferguson
On Jan 24, 2009, at 1:28 PM, Prasad Chakka wrote:
Down load the jar from here http://dev.mysql.com/downloads/connector/j/5.1.html
________________________________
From: Prasad Chakka <pr...@facebook.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:13:35 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
JPOX/Hive doesn't yet download the relevant client libraries based on the class name :) Drop mysql client jar in hive/lib. Also try setting org.jpox.autoCreateSchema to true.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:06:26 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
So I'm not really sure what the state of the database should be, I have a database called hive and the user has permissions to access it. I'm not sure which mysql client libraries need to be in the classpath, I assumed it was all rolled into the jpox stuff. Here is the complete error:
09/01/24 16:02:16 DEBUG exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:446)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:433)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:330)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:444)
... 16 more
On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
There could be many reasons for it. Could you post the stacktrace for the JDOFatal?
Some reasons could be
That mysql client libs are in the classpath.
That there is a database called 'hive' in mysql
That there is schema already created in that database (otherwise set autoCreateSchema to true) etc...
Check documentation for the many jpox options
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 12:48:27 -0800
To: <hi...@hadoop.apache.org>
Subject: Still can't get metastore working with mysql
Ok I still can't manage to get the metastore working with mysql after
all this time trying. The error is:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
The relevant config is:
<!-- Hadoop Setup -->
<property>
<name>hadoop.bin.path</name>
<value>/usr/local/hadoop/bin/hadoop</value>
<description>Path to hadoop binary. Assumes that by default we are
executing from hive</description>
</property>
<property>
<name>hadoop.config.dir</name>
<value>/usr/local/hadoop/conf</value>
<description>Path to hadoop configuration. Again assumes that by
default we are executing from hive/</description>
</property>
<!-- Hive Execution Parameters -->
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive-${user.name}</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>******</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>******</value>
</property>
<property>
<name>org.jpox.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>org.jpox.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.usefilestore</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.checkForDefaultDb</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.rawstore.impl</name>
<value>org.apache.hadoop.hive.metastore.ObjectStore</value>
<description>Name of the class that implements
org.apache.hadoop.hive.metastore.rawstore interface. This class is
used to store and retrieval of raw metadata objects such as table,
database</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server
or open a new metastore server in Hive Client JVM</description>
</property>
Re: Still can't get metastore working with mysql
Posted by Josh Ferguson <jo...@besquared.net>.
Joydeep Sen Sarma sent them out in the list last month when I was
first trying to get this working. I tried to modify them appropriately.
This whole "lazy" creation of schemas and tables seems like a really
bad way to do things. My configuration shouldn't be different for
something that is up and running and something that needs to be
created. I should just be able to define my connection parameters and
then have a command that migrates my database into the proper state
for use. Trying to figure out what magical set of esoteric
configuration parameters will tell this black box library to create my
schema is really frustrating.
Josh F.
On Jan 24, 2009, at 2:26 PM, Prasad Chakka wrote:
> Try setting fixedDataStore to false.
>
> BTB, where did you get the initial values from the config you
> posted? These values are for the store that is already running in
> production and you don’t expect any changes to the schema.
>
>
> From: Josh Ferguson <jo...@besquared.net>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 14:14:14 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Re: Still can't get metastore working with mysql
>
> That got me a little further, now I'm getting this error:
>
> FAILED: Error in metadata: MetaException(message:Got exception:
> javax.jdo.JDODataStoreException Required table missing : "`SERDES`"
> in Catalog "" Schema "". JPOX requires this table to perform its
> persistence operations. Either your MetaData is incorrect, or you
> need to enable "org.jpox.autoCreateTables")
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> In my hive-default.xml I turned on autoCreateTables and
> autoCreateSchema and it's still throwing this error back to me? Is
> there something else I need to do?
>
> Josh Ferguson
>
> On Jan 24, 2009, at 1:28 PM, Prasad Chakka wrote:
>
> Down load the jar from here http://dev.mysql.com/downloads/connector/j/5.1.html
>
>
>
> From: Prasad Chakka <pr...@facebook.com>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 13:13:35 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Re: Still can't get metastore working with mysql
>
> JPOX/Hive doesn’t yet download the relevant client libraries based
> on the class name :) Drop mysql client jar in hive/lib. Also try
> setting org.jpox.autoCreateSchema to true.
>
>
> From: Josh Ferguson <jo...@besquared.net>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 13:06:26 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Re: Still can't get metastore working with mysql
>
> So I'm not really sure what the state of the database should be, I
> have a database called hive and the user has permissions to access
> it. I'm not sure which mysql client libraries need to be in the
> classpath, I assumed it was all rolled into the jpox stuff. Here is
> the complete error:
>
> 09/01/24 16:02:16 DEBUG exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got
> exception: javax.jdo.JDOFatalInternalException Error creating
> transactional connection factory)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
> 446)
> at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:
> 433)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:
> 330)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:
> 174)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:
> 207)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun
> .reflect
> .NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun
> .reflect
> .DelegatingMethodAccessorImpl
> .invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
> Caused by: MetaException(message:Got exception:
> javax.jdo.JDOFatalInternalException Error creating transactional
> connection factory)
> at
> org
> .apache
> .hadoop
> .hive
> .metastore
> .MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
> at
> org
> .apache
> .hadoop
> .hive
> .metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
> 444)
> ... 16 more
>
>
> On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
>
>
> There could be many reasons for it. Could you post the stacktrace
> for the JDOFatal?
>
> Some reasons could be
> That mysql client libs are in the classpath.
> That there is a database called ‘hive’ in mysql
> That there is schema already created in that database (otherwise
> set autoCreateSchema to true) etc...
>
> Check documentation for the many jpox options
>
>
> From: Josh Ferguson <jo...@besquared.net>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 12:48:27 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Still can't get metastore working with mysql
>
> Ok I still can't manage to get the metastore working with mysql
> after
> all this time trying. The error is:
>
> FAILED: Error in metadata: MetaException(message:Got exception:
> javax.jdo.JDOFatalInternalException Error creating transactional
> connection factory)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> The relevant config is:
>
> <!-- Hadoop Setup -->
> <property>
> <name>hadoop.bin.path</name>
> <value>/usr/local/hadoop/bin/hadoop</value>
> <description>Path to hadoop binary. Assumes that by default we
> are
> executing from hive</description>
> </property>
>
> <property>
> <name>hadoop.config.dir</name>
> <value>/usr/local/hadoop/conf</value>
> <description>Path to hadoop configuration. Again assumes that by
> default we are executing from hive/</description>
> </property>
>
> <!-- Hive Execution Parameters -->
> <property>
> <name>hive.exec.scratchdir</name>
> <value>/tmp/hive-${user.name}</value>
> <description>Scratch space for Hive jobs</description>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://localhost/hive</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>******</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>******</value>
> </property>
>
> <property>
> <name>org.jpox.autoCreateSchema</name>
> <value>false</value>
> </property>
>
> <property>
> <name>org.jpox.fixedDatastore</name>
> <value>true</value>
> </property>
>
> <property>
> <name>hive.metastore.usefilestore</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.metastore.checkForDefaultDb</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.metastore.rawstore.impl</name>
> <value>org.apache.hadoop.hive.metastore.ObjectStore</value>
> <description>Name of the class that implements
> org.apache.hadoop.hive.metastore.rawstore interface. This class is
> used to store and retrieval of raw metadata objects such as table,
> database</description>
> </property>
>
> <property>
> <name>hive.metastore.local</name>
> <value>true</value>
> <description>controls whether to connect to remove metastore
> server
> or open a new metastore server in Hive Client JVM</description>
> </property>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Still can't get metastore working with mysql
Posted by Prasad Chakka <pr...@facebook.com>.
Try setting fixedDataStore to false.
BTB, where did you get the initial values from the config you posted? These values are for the store that is already running in production and you don't expect any changes to the schema.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 14:14:14 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
That got me a little further, now I'm getting this error:
FAILED: Error in metadata: MetaException(message:Got exception: javax.jdo.JDODataStoreException Required table missing : "`SERDES`" in Catalog "" Schema "". JPOX requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "org.jpox.autoCreateTables")
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
In my hive-default.xml I turned on autoCreateTables and autoCreateSchema and it's still throwing this error back to me? Is there something else I need to do?
Josh Ferguson
On Jan 24, 2009, at 1:28 PM, Prasad Chakka wrote:
Down load the jar from here http://dev.mysql.com/downloads/connector/j/5.1.html
________________________________
From: Prasad Chakka <pr...@facebook.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:13:35 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
JPOX/Hive doesn't yet download the relevant client libraries based on the class name :) Drop mysql client jar in hive/lib. Also try setting org.jpox.autoCreateSchema to true.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:06:26 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
So I'm not really sure what the state of the database should be, I have a database called hive and the user has permissions to access it. I'm not sure which mysql client libraries need to be in the classpath, I assumed it was all rolled into the jpox stuff. Here is the complete error:
09/01/24 16:02:16 DEBUG exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:446)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:433)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:330)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:444)
... 16 more
On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
There could be many reasons for it. Could you post the stacktrace for the JDOFatal?
Some reasons could be
That mysql client libs are in the classpath.
That there is a database called 'hive' in mysql
That there is schema already created in that database (otherwise set autoCreateSchema to true) etc...
Check documentation for the many jpox options
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 12:48:27 -0800
To: <hi...@hadoop.apache.org>
Subject: Still can't get metastore working with mysql
Ok I still can't manage to get the metastore working with mysql after
all this time trying. The error is:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
The relevant config is:
<!-- Hadoop Setup -->
<property>
<name>hadoop.bin.path</name>
<value>/usr/local/hadoop/bin/hadoop</value>
<description>Path to hadoop binary. Assumes that by default we are
executing from hive</description>
</property>
<property>
<name>hadoop.config.dir</name>
<value>/usr/local/hadoop/conf</value>
<description>Path to hadoop configuration. Again assumes that by
default we are executing from hive/</description>
</property>
<!-- Hive Execution Parameters -->
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive-${user.name}</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>******</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>******</value>
</property>
<property>
<name>org.jpox.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>org.jpox.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.usefilestore</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.checkForDefaultDb</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.rawstore.impl</name>
<value>org.apache.hadoop.hive.metastore.ObjectStore</value>
<description>Name of the class that implements
org.apache.hadoop.hive.metastore.rawstore interface. This class is
used to store and retrieval of raw metadata objects such as table,
database</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server
or open a new metastore server in Hive Client JVM</description>
</property>
Re: Still can't get metastore working with mysql
Posted by Josh Ferguson <jo...@besquared.net>.
That got me a little further, now I'm getting this error:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDODataStoreException Required table missing : "`SERDES`" in
Catalog "" Schema "". JPOX requires this table to perform its
persistence operations. Either your MetaData is incorrect, or you need
to enable "org.jpox.autoCreateTables")
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
In my hive-default.xml I turned on autoCreateTables and
autoCreateSchema and it's still throwing this error back to me? Is
there something else I need to do?
Josh Ferguson
On Jan 24, 2009, at 1:28 PM, Prasad Chakka wrote:
> Down load the jar from here http://dev.mysql.com/downloads/connector/j/5.1.html
>
>
> From: Prasad Chakka <pr...@facebook.com>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 13:13:35 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Re: Still can't get metastore working with mysql
>
> JPOX/Hive doesn’t yet download the relevant client libraries based
> on the class name :) Drop mysql client jar in hive/lib. Also try
> setting org.jpox.autoCreateSchema to true.
>
> From: Josh Ferguson <jo...@besquared.net>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 13:06:26 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Re: Still can't get metastore working with mysql
>
> So I'm not really sure what the state of the database should be, I
> have a database called hive and the user has permissions to access
> it. I'm not sure which mysql client libraries need to be in the
> classpath, I assumed it was all rolled into the jpox stuff. Here is
> the complete error:
>
> 09/01/24 16:02:16 DEBUG exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got
> exception: javax.jdo.JDOFatalInternalException Error creating
> transactional connection factory)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
> 446)
> at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:433)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:330)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:
> 207)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun
> .reflect
> .NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun
> .reflect
> .DelegatingMethodAccessorImpl
> .invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
> Caused by: MetaException(message:Got exception:
> javax.jdo.JDOFatalInternalException Error creating transactional
> connection factory)
> at
> org
> .apache
> .hadoop
> .hive
> .metastore
> .MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
> at
> org
> .apache
> .hadoop
> .hive
> .metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
> 444)
> ... 16 more
>
>
> On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
>
> There could be many reasons for it. Could you post the stacktrace
> for the JDOFatal?
>
> Some reasons could be
> That mysql client libs are in the classpath.
> That there is a database called ‘hive’ in mysql
> That there is schema already created in that database (otherwise
> set autoCreateSchema to true) etc...
>
> Check documentation for the many jpox options
>
> From: Josh Ferguson <jo...@besquared.net>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 12:48:27 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Still can't get metastore working with mysql
>
> Ok I still can't manage to get the metastore working with mysql after
> all this time trying. The error is:
>
> FAILED: Error in metadata: MetaException(message:Got exception:
> javax.jdo.JDOFatalInternalException Error creating transactional
> connection factory)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> The relevant config is:
>
> <!-- Hadoop Setup -->
> <property>
> <name>hadoop.bin.path</name>
> <value>/usr/local/hadoop/bin/hadoop</value>
> <description>Path to hadoop binary. Assumes that by default we are
> executing from hive</description>
> </property>
>
> <property>
> <name>hadoop.config.dir</name>
> <value>/usr/local/hadoop/conf</value>
> <description>Path to hadoop configuration. Again assumes that by
> default we are executing from hive/</description>
> </property>
>
> <!-- Hive Execution Parameters -->
> <property>
> <name>hive.exec.scratchdir</name>
> <value>/tmp/hive-${user.name}</value>
> <description>Scratch space for Hive jobs</description>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://localhost/hive</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>******</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>******</value>
> </property>
>
> <property>
> <name>org.jpox.autoCreateSchema</name>
> <value>false</value>
> </property>
>
> <property>
> <name>org.jpox.fixedDatastore</name>
> <value>true</value>
> </property>
>
> <property>
> <name>hive.metastore.usefilestore</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.metastore.checkForDefaultDb</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.metastore.rawstore.impl</name>
> <value>org.apache.hadoop.hive.metastore.ObjectStore</value>
> <description>Name of the class that implements
> org.apache.hadoop.hive.metastore.rawstore interface. This class is
> used to store and retrieval of raw metadata objects such as table,
> database</description>
> </property>
>
> <property>
> <name>hive.metastore.local</name>
> <value>true</value>
> <description>controls whether to connect to remove metastore
> server
> or open a new metastore server in Hive Client JVM</description>
> </property>
>
>
>
>
>
>
>
Re: Still can't get metastore working with mysql
Posted by Prasad Chakka <pr...@facebook.com>.
Down load the jar from here http://dev.mysql.com/downloads/connector/j/5.1.html
________________________________
From: Prasad Chakka <pr...@facebook.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:13:35 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
JPOX/Hive doesn't yet download the relevant client libraries based on the class name :) Drop mysql client jar in hive/lib. Also try setting org.jpox.autoCreateSchema to true.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:06:26 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
So I'm not really sure what the state of the database should be, I have a database called hive and the user has permissions to access it. I'm not sure which mysql client libraries need to be in the classpath, I assumed it was all rolled into the jpox stuff. Here is the complete error:
09/01/24 16:02:16 DEBUG exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:446)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:433)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:330)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:444)
... 16 more
On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
There could be many reasons for it. Could you post the stacktrace for the JDOFatal?
Some reasons could be
That mysql client libs are in the classpath.
That there is a database called 'hive' in mysql
That there is schema already created in that database (otherwise set autoCreateSchema to true) etc...
Check documentation for the many jpox options
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 12:48:27 -0800
To: <hi...@hadoop.apache.org>
Subject: Still can't get metastore working with mysql
Ok I still can't manage to get the metastore working with mysql after
all this time trying. The error is:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
The relevant config is:
<!-- Hadoop Setup -->
<property>
<name>hadoop.bin.path</name>
<value>/usr/local/hadoop/bin/hadoop</value>
<description>Path to hadoop binary. Assumes that by default we are
executing from hive</description>
</property>
<property>
<name>hadoop.config.dir</name>
<value>/usr/local/hadoop/conf</value>
<description>Path to hadoop configuration. Again assumes that by
default we are executing from hive/</description>
</property>
<!-- Hive Execution Parameters -->
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive-${user.name}</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>******</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>******</value>
</property>
<property>
<name>org.jpox.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>org.jpox.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.usefilestore</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.checkForDefaultDb</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.rawstore.impl</name>
<value>org.apache.hadoop.hive.metastore.ObjectStore</value>
<description>Name of the class that implements
org.apache.hadoop.hive.metastore.rawstore interface. This class is
used to store and retrieval of raw metadata objects such as table,
database</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server
or open a new metastore server in Hive Client JVM</description>
</property>
Re: Still can't get metastore working with mysql
Posted by Prasad Chakka <pr...@facebook.com>.
JPOX/Hive doesn't yet download the relevant client libraries based on the class name :) Drop mysql client jar in hive/lib. Also try setting org.jpox.autoCreateSchema to true.
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 13:06:26 -0800
To: <hi...@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql
So I'm not really sure what the state of the database should be, I have a database called hive and the user has permissions to access it. I'm not sure which mysql client libraries need to be in the classpath, I assumed it was all rolled into the jpox stuff. Here is the complete error:
09/01/24 16:02:16 DEBUG exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:446)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:433)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:330)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional connection factory)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:444)
... 16 more
On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
There could be many reasons for it. Could you post the stacktrace for the JDOFatal?
Some reasons could be
That mysql client libs are in the classpath.
That there is a database called 'hive' in mysql
That there is schema already created in that database (otherwise set autoCreateSchema to true) etc...
Check documentation for the many jpox options
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 12:48:27 -0800
To: <hi...@hadoop.apache.org>
Subject: Still can't get metastore working with mysql
Ok I still can't manage to get the metastore working with mysql after
all this time trying. The error is:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
The relevant config is:
<!-- Hadoop Setup -->
<property>
<name>hadoop.bin.path</name>
<value>/usr/local/hadoop/bin/hadoop</value>
<description>Path to hadoop binary. Assumes that by default we are
executing from hive</description>
</property>
<property>
<name>hadoop.config.dir</name>
<value>/usr/local/hadoop/conf</value>
<description>Path to hadoop configuration. Again assumes that by
default we are executing from hive/</description>
</property>
<!-- Hive Execution Parameters -->
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive-${user.name}</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>******</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>******</value>
</property>
<property>
<name>org.jpox.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>org.jpox.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.usefilestore</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.checkForDefaultDb</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.rawstore.impl</name>
<value>org.apache.hadoop.hive.metastore.ObjectStore</value>
<description>Name of the class that implements
org.apache.hadoop.hive.metastore.rawstore interface. This class is
used to store and retrieval of raw metadata objects such as table,
database</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server
or open a new metastore server in Hive Client JVM</description>
</property>
Re: Still can't get metastore working with mysql
Posted by Josh Ferguson <jo...@besquared.net>.
So I'm not really sure what the state of the database should be, I
have a database called hive and the user has permissions to access it.
I'm not sure which mysql client libraries need to be in the classpath,
I assumed it was all rolled into the jpox stuff. Here is the complete
error:
09/01/24 16:02:16 DEBUG exec.DDLTask:
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got
exception: javax.jdo.JDOFatalInternalException Error creating
transactional connection factory)
at
org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
446)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:433)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:330)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun
.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
at
org
.apache
.hadoop
.hive
.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:
543)
at
org
.apache
.hadoop
.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:
451)
at
org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
444)
... 16 more
On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
> There could be many reasons for it. Could you post the stacktrace
> for the JDOFatal?
>
> Some reasons could be
> That mysql client libs are in the classpath.
> That there is a database called ‘hive’ in mysql
> That there is schema already created in that database (otherwise set
> autoCreateSchema to true) etc...
>
> Check documentation for the many jpox options
> From: Josh Ferguson <jo...@besquared.net>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Sat, 24 Jan 2009 12:48:27 -0800
> To: <hi...@hadoop.apache.org>
> Subject: Still can't get metastore working with mysql
>
> Ok I still can't manage to get the metastore working with mysql after
> all this time trying. The error is:
>
> FAILED: Error in metadata: MetaException(message:Got exception:
> javax.jdo.JDOFatalInternalException Error creating transactional
> connection factory)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> The relevant config is:
>
> <!-- Hadoop Setup -->
> <property>
> <name>hadoop.bin.path</name>
> <value>/usr/local/hadoop/bin/hadoop</value>
> <description>Path to hadoop binary. Assumes that by default we are
> executing from hive</description>
> </property>
>
> <property>
> <name>hadoop.config.dir</name>
> <value>/usr/local/hadoop/conf</value>
> <description>Path to hadoop configuration. Again assumes that by
> default we are executing from hive/</description>
> </property>
>
> <!-- Hive Execution Parameters -->
> <property>
> <name>hive.exec.scratchdir</name>
> <value>/tmp/hive-${user.name}</value>
> <description>Scratch space for Hive jobs</description>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://localhost/hive</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>******</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>******</value>
> </property>
>
> <property>
> <name>org.jpox.autoCreateSchema</name>
> <value>false</value>
> </property>
>
> <property>
> <name>org.jpox.fixedDatastore</name>
> <value>true</value>
> </property>
>
> <property>
> <name>hive.metastore.usefilestore</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.metastore.checkForDefaultDb</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.metastore.rawstore.impl</name>
> <value>org.apache.hadoop.hive.metastore.ObjectStore</value>
> <description>Name of the class that implements
> org.apache.hadoop.hive.metastore.rawstore interface. This class is
> used to store and retrieval of raw metadata objects such as table,
> database</description>
> </property>
>
> <property>
> <name>hive.metastore.local</name>
> <value>true</value>
> <description>controls whether to connect to remove metastore server
> or open a new metastore server in Hive Client JVM</description>
> </property>
>
>
Re: Still can't get metastore working with mysql
Posted by Prasad Chakka <pr...@facebook.com>.
There could be many reasons for it. Could you post the stacktrace for the JDOFatal?
Some reasons could be
That mysql client libs are in the classpath.
That there is a database called 'hive' in mysql
That there is schema already created in that database (otherwise set autoCreateSchema to true) etc...
Check documentation for the many jpox options
________________________________
From: Josh Ferguson <jo...@besquared.net>
Reply-To: <hi...@hadoop.apache.org>
Date: Sat, 24 Jan 2009 12:48:27 -0800
To: <hi...@hadoop.apache.org>
Subject: Still can't get metastore working with mysql
Ok I still can't manage to get the metastore working with mysql after
all this time trying. The error is:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
The relevant config is:
<!-- Hadoop Setup -->
<property>
<name>hadoop.bin.path</name>
<value>/usr/local/hadoop/bin/hadoop</value>
<description>Path to hadoop binary. Assumes that by default we are
executing from hive</description>
</property>
<property>
<name>hadoop.config.dir</name>
<value>/usr/local/hadoop/conf</value>
<description>Path to hadoop configuration. Again assumes that by
default we are executing from hive/</description>
</property>
<!-- Hive Execution Parameters -->
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive-${user.name}</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>******</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>******</value>
</property>
<property>
<name>org.jpox.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>org.jpox.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.usefilestore</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.checkForDefaultDb</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.rawstore.impl</name>
<value>org.apache.hadoop.hive.metastore.ObjectStore</value>
<description>Name of the class that implements
org.apache.hadoop.hive.metastore.rawstore interface. This class is
used to store and retrieval of raw metadata objects such as table,
database</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server
or open a new metastore server in Hive Client JVM</description>
</property>