You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by bharath vissapragada <bh...@gmail.com> on 2011/08/19 07:08:26 UTC
Re: Setting up stats database[SOLVED]
Hi,
I solved this by placing the jar in ${java_home}/jre/lib and
${java_home}/jre/lib/ext . This is the workaround whenever jdbc drivers wont
work. The same thing worked here too. (I hope it works with your postgres
too). I am still wondering why hive didn't recognize it in the classpath.
Also there is some parsing problem in my connection string and it is getting
terminated at ";" in
jdbc:mysql://<ip>:3306/TempStatsStore&user=<name>&password=<pwd>. I
got it worked by adding 2 properties stats.username and stats.password just
like the metastore db user and password, and replaced the
conn = DriverManager.getConnection(connectionString) with conn =
DriverManager.getConnection(connectionString,<uname>,<pwd>) by reading them
from Conf variable inside JDBCStatsPublisher class.
Is this worth filing a JIRA or Am I the only one facing this problem?
Thanks
On Fri, Aug 19, 2011 at 8:05 AM, wd <wd...@wdicc.com> wrote:
> Maybe you should use 'hive.stats.jdbcdriver="org.apache.mysql.jdbc.EmbeddedDriver"'
> settings?
>
> via
> http://mail-archives.apache.org/mod_mbox/hive-user/201103.mbox/%3C42360B00-72EC-437A-9D95-93F3AD9F1B7D@fb.com%3E
>
> On Fri, Aug 19, 2011 at 5:45 AM, bharath vissapragada <
> bharathvissapragada1990@gmail.com> wrote:
>
>> Hi,
>>
>> Iam also getting the same error. However I am using mysql for stats.
>>
>> The thing is I configured mysql for metastore and it works fine and all
>> the metadata gets populated normally. When the metastore classes can find
>> the mysql jar in the class path , why cant the stats publisher find it. I
>> looked at the stats source and everything looks fine.
>>
>> My conn string is :
>> jdbc:mysql://<ip>:3306/TempStatsStore&user=<name>&password=<pwd>.
>>
>> Am I missing something?
>>
>> Thanks
>>
>>
>>
>>
>> On Thu, Aug 18, 2011 at 8:19 AM, wd <wd...@wdicc.com> wrote:
>>
>>> The error in log is 'java.lang.ClassNotFoundException:
>>> org.postgresql.Driver', not can't connect or user name or password error.
>>>
>>>
>>> On Wed, Aug 17, 2011 at 3:53 PM, Jander g <ja...@gmail.com> wrote:
>>>
>>>> Hi,wd
>>>>
>>>> You should configure "hive.stats.dbconnectionstring" as follows.
>>>>
>>>>
>>>> <property>
>>>> <name>hive.stats.dbconnectionstring</name>
>>>>
>>>> <value>jdbc:postgresql://localhost/hive_statsdb?createDatabaseIfNotExist=true&user=hive&password=pwd</value>
>>>>
>>>> <description>The default connection string for the database that
>>>> stores temporary hive statistics.</description>
>>>> </property>
>>>>
>>>> Regards,
>>>>
>>>> Jander.
>>>>
>>>>
>>>> On Mon, Aug 15, 2011 at 3:09 PM, wd <wd...@wdicc.com> wrote:
>>>>
>>>>> hi,
>>>>>
>>>>> I'm try to use postgres as stats database. And made following settings
>>>>> in hive-site.xml
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>hive.stats.dbclass</name>
>>>>> <value>jdbc:postgresql</value>
>>>>> <description>The default database that stores temporary hive
>>>>> statistics.</description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>hive.stats.autogather</name>
>>>>> <value>true</value>
>>>>> <description>A flag to gather statistics automatically during the
>>>>> INSERT OVERWRITE command.</description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>hive.stats.jdbcdriver</name>
>>>>> <value>org.postgresql.Driver</value>
>>>>> <description>The JDBC driver for the database that stores temporary
>>>>> hive statistics.</description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>hive.stats.dbconnectionstring</name>
>>>>>
>>>>> <value>jdbc:postgresql://localhost/hive_statsdb?createDatabaseIfNotExist=true;user=hive;password=pwd</value>
>>>>> <description>The default connection string for the database that
>>>>> stores temporary hive statistics.</description>
>>>>> </property>
>>>>>
>>>>> I use postgres as hive meta database, so there is a
>>>>> postgresql-9.0-801.jdbc4.jar file in lib.
>>>>>
>>>>> After run 'analyse table t1 partitions(dt) comput statistics;' in hive
>>>>> cli, it will output some stats info in cli, but nothing in db. And I
>>>>> can found there is the flowing errors
>>>>>
>>>>> 1-08-15 14:54:54,767 INFO
>>>>> org.apache.hadoop.hive.ql.exec.TableScanOperator: Stats Gathering
>>>>> found a new partition spec = dt=20110805
>>>>> 2011-08-15 14:54:54,767 INFO
>>>>> org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarding 1 rows
>>>>> 2011-08-15 14:54:54,767 INFO ExecMapper: ExecMapper: processing 1
>>>>> rows: used memory = 39953640
>>>>> 2011-08-15 14:54:54,768 INFO
>>>>> org.apache.hadoop.hive.ql.exec.MapOperator: 1 finished. closing...
>>>>> 2011-08-15 14:54:54,768 INFO
>>>>> org.apache.hadoop.hive.ql.exec.MapOperator: 1 forwarded 2 rows
>>>>> 2011-08-15 14:54:54,768 INFO
>>>>> org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0
>>>>> 2011-08-15 14:54:54,768 INFO
>>>>> org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 finished.
>>>>> closing...
>>>>> 2011-08-15 14:54:54,768 INFO
>>>>> org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarded 2 rows
>>>>> 2011-08-15 14:54:54,772 ERROR
>>>>> org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher: Error during
>>>>> JDBC connection to
>>>>>
>>>>> jdbc:postgresql://localhost/hive_statsdb?createDatabaseIfNotExist=true;user=hive;password=pwd.
>>>>> java.lang.ClassNotFoundException: org.postgresql.Driver
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>> at java.lang.Class.forName0(Native Method)
>>>>> at java.lang.Class.forName(Class.java:169)
>>>>> at
>>>>> org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.connect(JDBCStatsPublisher.java:55)
>>>>> at
>>>>> org.apache.hadoop.hive.ql.exec.TableScanOperator.publishStats(TableScanOperator.java:202)
>>>>> at
>>>>> org.apache.hadoop.hive.ql.exec.TableScanOperator.closeOp(TableScanOperator.java:164)
>>>>> at
>>>>> org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:557)
>>>>> at
>>>>> org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
>>>>> at
>>>>> org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
>>>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
>>>>> at
>>>>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>> 2011-08-15 14:54:54,774 INFO
>>>>> org.apache.hadoop.hive.ql.exec.TableScanOperator: StatsPublishing
>>>>> error: cannot connect to database.
>>>>> 2011-08-15 14:54:54,774 INFO
>>>>> org.apache.hadoop.hive.ql.exec.MapOperator: 1 Close done
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Jander
>>>>
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Bharath .V
>> w:http://researchweb.iiit.ac.in/~bharath.v
>>
>
>
--
Regards,
Bharath .V
w:http://researchweb.iiit.ac.in/~bharath.v