You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Lefty Leverenz <le...@gmail.com> on 2014/01/15 10:23:09 UTC
Re: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
Nirmal, did you get the help you needed (or figure it out yourself)?
Can you please let me know what we mean by *schematool -dbType mysql
> –initSchema ?*
>
>
>
> Also the **order** in which these commands needs to be invoked:
>
> 1. *schematool -dbType mysql -initSchema*
>
> *2. **schematool -dbType mysql -upgradeSchemaFrom 0.9.0*
>
-- Lefty
On Sun, Dec 29, 2013 at 12:14 AM, Lefty Leverenz <le...@gmail.com>wrote:
> ... and by "Don't all talk at once" what I meant was "somebody please
> help."
>
> -- Lefty
>
>
> On Fri, Dec 20, 2013 at 3:39 PM, Lefty Leverenz <le...@gmail.com>wrote:
>
>> Sure thing, if by "you" you mean "not Lefty but somebody who actually
>> knows about this." (Don't all talk at once now....)
>>
>> -- Lefty
>>
>>
>> On Fri, Dec 20, 2013 at 3:40 AM, Nirmal Kumar <nirmal.kumar@impetus.co.in
>> > wrote:
>>
>>> Hi Lefty,
>>>
>>>
>>>
>>> Thanks for the useful info about the schematool.
>>>
>>>
>>>
>>> I am doing a schema upgrade of a MySQL backed Hive MetaStore instance
>>> from Hive 0.9 to Hive 0.12.
>>>
>>>
>>>
>>> *hive-site.xml*
>>>
>>>
>>>
>>> <configuration>
>>>
>>> <property>
>>>
>>> <name>javax.jdo.option.ConnectionURL</name>
>>>
>>>
>>> <value>jdbc:mysql://localhost:3306/hive_meta_store_mysql4_schematool?createDatabaseIfNotExist=true</value>
>>>
>>> <description>JDBC connect string for a JDBC metastore</description>
>>>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>>
>>> <name>javax.jdo.option.ConnectionDriverName</name>
>>>
>>> <value>com.mysql.jdbc.Driver</value>
>>>
>>> <description>Driver class name for a JDBC metastore</description>
>>>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>>
>>> <name>javax.jdo.option.ConnectionUserName</name>
>>>
>>> <value>root</value>
>>>
>>> <description>username to use against metastore database</description>
>>>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>>
>>> <name>javax.jdo.option.ConnectionPassword</name>
>>>
>>> <value>rapid</value>
>>>
>>> <description>password to use against metastore database,single space
>>> means no password</description>
>>>
>>> </property>
>>>
>>>
>>>
>>> *<property>*
>>>
>>> * <name>hive.metastore.schema.verification</name>*
>>>
>>> * <value>true</value>*
>>>
>>> * <description>Default false and metastore to implicitly write the
>>> schema version if it's not matching</description>*
>>>
>>> *</property>*
>>>
>>> </configuration>
>>>
>>>
>>>
>>>
>>>
>>> $ schematool -help
>>>
>>> usage: schemaTool
>>>
>>> -dbType <databaseType> Metastore database type
>>>
>>> -dryRun List SQL scripts (no execute)
>>>
>>> -help Print this message
>>>
>>> -info Show config and schema details
>>>
>>> -initSchema Schema initialization
>>>
>>> -initSchemaTo <initTo> Schema initialization to a version
>>>
>>> -passWord <password> Override config file password
>>>
>>> -upgradeSchema Schema upgrade
>>>
>>> -upgradeSchemaFrom <upgradeFrom> Schema upgrade from a version
>>>
>>> -userName <user> Override config file user name
>>>
>>> -verbose Only print SQL statements
>>>
>>>
>>>
>>> I am using the following commands:
>>>
>>>
>>>
>>> [cloud@Impetus-942 hadoop_migration]$ *schematool -dbType mysql
>>> -upgradeSchemaFrom 0.9.0 -dryRun*
>>>
>>> 13/12/20 17:58:59 WARN conf.Configuration: mapred.max.split.size is
>>> deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
>>>
>>> 13/12/20 17:58:59 WARN conf.Configuration: mapred.min.split.size is
>>> deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
>>>
>>> 13/12/20 17:58:59 WARN conf.Configuration:
>>> mapred.min.split.size.per.rack is deprecated. Instead, use
>>> mapreduce.input.fileinputformat.split.minsize.per.rack
>>>
>>> 13/12/20 17:58:59 WARN conf.Configuration:
>>> mapred.min.split.size.per.node is deprecated. Instead, use
>>> mapreduce.input.fileinputformat.split.minsize.per.node
>>>
>>> 13/12/20 17:58:59 WARN conf.Configuration: mapred.reduce.tasks is
>>> deprecated. Instead, use mapreduce.job.reduces
>>>
>>> 13/12/20 17:58:59 WARN conf.Configuration:
>>> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
>>> mapreduce.reduce.speculative
>>>
>>> Metastore connection URL:
>>> jdbc:mysql://localhost:3306/hive_meta_store_mysql4_schematool?createDatabaseIfNotExist=true
>>>
>>> Metastore Connection Driver : com.mysql.jdbc.Driver
>>>
>>> Metastore connection User: root
>>>
>>> Starting upgrade metastore schema from version 0.9.0 to 0.12.0
>>>
>>> Upgrade script upgrade-0.9.0-to-0.10.0.mysql.sql
>>>
>>> Upgrade script upgrade-0.10.0-to-0.11.0.mysql.sql
>>>
>>> Upgrade script upgrade-0.11.0-to-0.12.0.mysql.sql
>>>
>>> schemaTool completeted
>>>
>>> [cloud@Impetus-942 hadoop_migration]$ *schematool -dbType mysql
>>> -upgradeSchemaFrom 0.9.0*
>>>
>>> 13/12/20 17:59:19 WARN conf.Configuration: mapred.max.split.size is
>>> deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
>>>
>>> 13/12/20 17:59:19 WARN conf.Configuration: mapred.min.split.size is
>>> deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
>>>
>>> 13/12/20 17:59:19 WARN conf.Configuration:
>>> mapred.min.split.size.per.rack is deprecated. Instead, use
>>> mapreduce.input.fileinputformat.split.minsize.per.rack
>>>
>>> 13/12/20 17:59:19 WARN conf.Configuration:
>>> mapred.min.split.size.per.node is deprecated. Instead, use
>>> mapreduce.input.fileinputformat.split.minsize.per.node
>>>
>>> 13/12/20 17:59:19 WARN conf.Configuration: mapred.reduce.tasks is
>>> deprecated. Instead, use mapreduce.job.reduces
>>>
>>> 13/12/20 17:59:19 WARN conf.Configuration:
>>> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
>>> mapreduce.reduce.speculative
>>>
>>> Metastore connection URL:
>>> jdbc:mysql://localhost:3306/hive_meta_store_mysql4_schematool?createDatabaseIfNotExist=true
>>>
>>> Metastore Connection Driver : com.mysql.jdbc.Driver
>>>
>>> Metastore connection User: root
>>>
>>> Starting upgrade metastore schema from version 0.9.0 to 0.12.0
>>>
>>> Upgrade script upgrade-0.9.0-to-0.10.0.mysql.sql
>>>
>>> Completed upgrade-0.9.0-to-0.10.0.mysql.sql
>>>
>>> Upgrade script upgrade-0.10.0-to-0.11.0.mysql.sql
>>>
>>> Completed upgrade-0.10.0-to-0.11.0.mysql.sql
>>>
>>> Upgrade script upgrade-0.11.0-to-0.12.0.mysql.sql
>>>
>>> Completed upgrade-0.11.0-to-0.12.0.mysql.sql
>>>
>>> schemaTool completeted
>>>
>>> [cloud@Impetus-942 hadoop_migration]$
>>>
>>>
>>>
>>> After this the following command gives me error:
>>>
>>>
>>>
>>> [cloud@Impetus-942 hadoop_migration]$ *schematool -dbType mysql
>>> -initSchema*
>>>
>>> 13/12/20 18:00:59 WARN conf.Configuration: mapred.max.split.size is
>>> deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
>>>
>>> 13/12/20 18:00:59 WARN conf.Configuration: mapred.min.split.size is
>>> deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
>>>
>>> 13/12/20 18:00:59 WARN conf.Configuration:
>>> mapred.min.split.size.per.rack is deprecated. Instead, use
>>> mapreduce.input.fileinputformat.split.minsize.per.rack
>>>
>>> 13/12/20 18:00:59 WARN conf.Configuration:
>>> mapred.min.split.size.per.node is deprecated. Instead, use
>>> mapreduce.input.fileinputformat.split.minsize.per.node
>>>
>>> 13/12/20 18:00:59 WARN conf.Configuration: mapred.reduce.tasks is
>>> deprecated. Instead, use mapreduce.job.reduces
>>>
>>> 13/12/20 18:00:59 WARN conf.Configuration:
>>> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
>>> mapreduce.reduce.speculative
>>>
>>> Metastore connection URL:
>>> jdbc:mysql://localhost:3306/hive_meta_store_mysql4_schematool?createDatabaseIfNotExist=true
>>>
>>> Metastore Connection Driver : com.mysql.jdbc.Driver
>>>
>>> Metastore connection User: root
>>>
>>> Starting metastore schema initialization to 0.12.0
>>>
>>> Initialization script hive-schema-0.12.0.mysql.sql
>>>
>>> Error: Duplicate entry '1' for key 'PRIMARY' (state=23000,code=1062)
>>>
>>> org.apache.hadoop.hive.metastore.HiveMetaException: Schema
>>> initialization FAILED! Metastore state would be inconsistent !!
>>>
>>> *** schemaTool failed ***
>>>
>>>
>>>
>>> Can you please let me know what we mean by *schematool -dbType mysql
>>> –initSchema ?*
>>>
>>>
>>>
>>> Also the **order** in which these commands needs to be invoked:
>>>
>>> 1. *schematool -dbType mysql -initSchema*
>>>
>>> *2. **schematool -dbType mysql -upgradeSchemaFrom 0.9.0*
>>>
>>>
>>>
>>> Thanks,
>>>
>>> -Nirmal
>>>
>>>
>>>
>>> *From:* Lefty Leverenz [mailto:leftyleverenz@gmail.com]
>>> *Sent:* Friday, December 20, 2013 6:32 AM
>>>
>>> *To:* user@hive.apache.org
>>> *Subject:* Re: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
>>>
>>>
>>>
>>> You should be aware of a new offline tool in Hive 0.12: The Hive
>>> Schema Tool<https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool#HiveSchemaTool-TheHiveSchemaTool>
>>> .
>>>
>>>
>>>
>>> This tool can be used to initialize the metastore schema for the current
>>> Hive version. It can also handle upgrading the schema from an older version
>>> to current.
>>>
>>>
>>> -- Lefty
>>>
>>>
>>>
>>> On Wed, Dec 18, 2013 at 9:09 AM, Nirmal Kumar <
>>> nirmal.kumar@impetus.co.in> wrote:
>>>
>>> Thanks Dima for pointing out the important info.
>>>
>>>
>>>
>>> I’m doing a similar migration from **Hive 0.9 to Hive 0.12** and
>>> looking for some steps that needs to be performed as part of the
>>> migrate\upgrade.
>>>
>>> Can you please share me the steps that you followed or referred?
>>>
>>>
>>>
>>> Also I’m looking for what else needs to be considered if one goes for a
>>> migration of newer version of Hive from an older version.
>>>
>>> I am assuming of some **backward compatibility** so that the existing
>>> queries that ran fine on the older version can run on the new version as
>>> well.
>>>
>>> This way I assume there won’t be any manual changes required in the
>>> queries.
>>>
>>>
>>>
>>> I agree with you though that the queries needs to be tested thoroughly
>>> on the new version as well. Also yes there can be some new bugs with the
>>> new version.
>>>
>>>
>>>
>>> -Nirmal
>>>
>>>
>>>
>>> *From:* Dima Machlin [mailto:Dima.Machlin@pursway.com]
>>> *Sent:* Wednesday, December 18, 2013 4:17 PM
>>>
>>>
>>> *To:* user@hive.apache.org
>>> *Subject:* RE: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
>>>
>>>
>>>
>>> Hi Nirmal,
>>>
>>> I recently performed a similar upgrade process from Hive 0.7 to Hive 0.10
>>>
>>> It seems that as some bugs in the 0.7 versions are solved in newer
>>> release, some new bugs appear.
>>>
>>>
>>>
>>> I’d recommend to setup a test environment running the newer version of
>>> Hive, and testing the SQLs you are running on the current.
>>>
>>> Then, compare the results in a one-to-one test checking that the data
>>> remains the same.
>>>
>>>
>>>
>>> From my experience I ran into 2 bugs in Hive 0.10 that don’t appear in
>>> Hive 0.7 and required making some changes in the SQLs
>>>
>>> The bugs are :
>>>
>>> 1) HIVE-5964 <https://issues.apache.org/jira/browse/HIVE-5964> – a
>>> very specific edge case causing hive to miss a filter (thus causing wrong
>>> results)
>>>
>>> 2) HIVE-5607 <https://issues.apache.org/jira/browse/HIVE-5607> –
>>> Hive failing to parse the “%” (mod) sign in certain conditions.
>>>
>>>
>>>
>>> In your case you might come up with different bugs or none at all, but
>>> if your environment is a production environment I’d recommend performing a
>>> full test before upgrading.
>>>
>>> Another option is to go over all the known bugs in v0.10 and see if they
>>> should affect your queries.
>>>
>>> By doing so, you will be taking the risk of encountering unknown bugs.
>>>
>>>
>>>
>>>
>>>
>>> *From:* Nirmal Kumar [mailto:nirmal.kumar@impetus.co.in<ni...@impetus.co.in>]
>>>
>>> *Sent:* Tuesday, December 17, 2013 8:49 PM
>>> *To:* user@hive.apache.org
>>> *Subject:* RE: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
>>>
>>>
>>>
>>> Thanks Arun,
>>>
>>>
>>>
>>> I will definitely move to the stable Hadoop-2.2.0 and Hive-0.12 versions.
>>>
>>>
>>>
>>> All I want to know is whether upgrading\migrating from Hive 0.7/0.8/0.9
>>> to Hive 0.12 requires some changes in the Hive queries or not? If yes where
>>> can I get the info?
>>>
>>>
>>>
>>> -Nirmal
>>>
>>>
>>>
>>> *From:* Arun C Murthy [mailto:acm@hortonworks.com <ac...@hortonworks.com>]
>>>
>>> *Sent:* Wednesday, December 18, 2013 12:07 AM
>>> *To:* user@hive.apache.org
>>> *Subject:* Re: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
>>>
>>>
>>>
>>> I'd strongly encourage you to go straight to hadoop-2.2.0 (the hadoop-2
>>> GA release).
>>>
>>>
>>>
>>> Also, I'd look at hive-0.12 which has been well tested against
>>> hadoop-2.2.0.
>>>
>>>
>>>
>>> thanks,
>>>
>>> Arun
>>>
>>>
>>>
>>> On Dec 17, 2013, at 12:43 AM, Nirmal Kumar <ni...@impetus.co.in>
>>> wrote:
>>>
>>>
>>>
>>> Hi All,
>>>
>>>
>>>
>>> I have migrated from *Apache Hadoop 1.2.0* to *Apache Hadoop
>>> 2.1.0-beta *and now I want to migrate the Hive version(0.10.0 +) as
>>> well from an older version(0.7/0/8/0.9).
>>>
>>>
>>>
>>> This is since the support for Hive to work on YARN in Hadoop 0.23 and
>>> 2.x releases has been supported since 0.10.0.
>>>
>>>
>>>
>>> I wanted to know that if migrating from Hive 0.7/0.8/0.9 to Hive 0.10.0
>>> requires some changes in the Hive queries or not. If the queries do need
>>> some changes then where can I get the info.?
>>>
>>>
>>>
>>> Also are there some steps for migrating Hive 0.7/0.8/0.9 to Hive 0.10.0 ?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> -Nirmal
>>>
>>>
>>> ------------------------------
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> NOTE: This message may contain information that is confidential,
>>> proprietary, privileged or otherwise protected by law. The message is
>>> intended solely for the named addressee. If received in error, please
>>> destroy and notify the sender. Any use of this email is prohibited when
>>> received in error. Impetus does not represent, warrant and/or guarantee,
>>> that the integrity of this communication has been maintained nor that the
>>> communication is free of errors, virus, interception or interference.
>>>
>>>
>>>
>>> --
>>>
>>> Arun C. Murthy
>>>
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>>
>>> ------------------------------
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> NOTE: This message may contain information that is confidential,
>>> proprietary, privileged or otherwise protected by law. The message is
>>> intended solely for the named addressee. If received in error, please
>>> destroy and notify the sender. Any use of this email is prohibited when
>>> received in error. Impetus does not represent, warrant and/or guarantee,
>>> that the integrity of this communication has been maintained nor that the
>>> communication is free of errors, virus, interception or interference.
>>>
>>>
>>>
>>>
>>>
>>> ************************************************************************************
>>> This footnote confirms that this email message has been scanned by
>>> PineApp Mail-SeCure for the presence of malicious code, vandals &
>>> computer viruses.
>>>
>>> ************************************************************************************
>>>
>>>
>>> ------------------------------
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> NOTE: This message may contain information that is confidential,
>>> proprietary, privileged or otherwise protected by law. The message is
>>> intended solely for the named addressee. If received in error, please
>>> destroy and notify the sender. Any use of this email is prohibited when
>>> received in error. Impetus does not represent, warrant and/or guarantee,
>>> that the integrity of this communication has been maintained nor that the
>>> communication is free of errors, virus, interception or interference.
>>>
>>>
>>>
>>> ------------------------------
>>>
>>>
>>>
>>>
>>>
>>>
>>> NOTE: This message may contain information that is confidential,
>>> proprietary, privileged or otherwise protected by law. The message is
>>> intended solely for the named addressee. If received in error, please
>>> destroy and notify the sender. Any use of this email is prohibited when
>>> received in error. Impetus does not represent, warrant and/or guarantee,
>>> that the integrity of this communication has been maintained nor that the
>>> communication is free of errors, virus, interception or interference.
>>>
>>
>>
>
RE: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Well Lefty,
I just used the command:
schematool -dbType mysql -upgradeSchemaFrom 0.9.0
I didn't used the schematool -dbType mysql -initSchema
I still didn't understood the order and meaning of the 2 commands.
Thanks,
-Nirmal
From: Lefty Leverenz [mailto:leftyleverenz@gmail.com]
Sent: Wednesday, January 15, 2014 2:53 PM
To: user@hive.apache.org
Subject: Re: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
Nirmal, did you get the help you needed (or figure it out yourself)?
Can you please let me know what we mean by schematool -dbType mysql -initSchema ?
Also the *order* in which these commands needs to be invoked:
1. schematool -dbType mysql -initSchema
2. schematool -dbType mysql -upgradeSchemaFrom 0.9.0
-- Lefty
On Sun, Dec 29, 2013 at 12:14 AM, Lefty Leverenz <le...@gmail.com>> wrote:
... and by "Don't all talk at once" what I meant was "somebody please help."
-- Lefty
On Fri, Dec 20, 2013 at 3:39 PM, Lefty Leverenz <le...@gmail.com>> wrote:
Sure thing, if by "you" you mean "not Lefty but somebody who actually knows about this." (Don't all talk at once now....)
-- Lefty
On Fri, Dec 20, 2013 at 3:40 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi Lefty,
Thanks for the useful info about the schematool.
I am doing a schema upgrade of a MySQL backed Hive MetaStore instance from Hive 0.9 to Hive 0.12.
hive-site.xml
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive_meta_store_mysql4_schematool?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>rapid</value>
<description>password to use against metastore database,single space means no password</description>
</property>
<property>
<name>hive.metastore.schema.verification</name>
<value>true</value>
<description>Default false and metastore to implicitly write the schema version if it's not matching</description>
</property>
</configuration>
$ schematool -help
usage: schemaTool
-dbType <databaseType> Metastore database type
-dryRun List SQL scripts (no execute)
-help Print this message
-info Show config and schema details
-initSchema Schema initialization
-initSchemaTo <initTo> Schema initialization to a version
-passWord <password> Override config file password
-upgradeSchema Schema upgrade
-upgradeSchemaFrom <upgradeFrom> Schema upgrade from a version
-userName <user> Override config file user name
-verbose Only print SQL statements
I am using the following commands:
[cloud@Impetus-942 hadoop_migration]$ schematool -dbType mysql -upgradeSchemaFrom 0.9.0 -dryRun
13/12/20 17:58:59 WARN conf.Configuration: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
13/12/20 17:58:59 WARN conf.Configuration: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
13/12/20 17:58:59 WARN conf.Configuration: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
13/12/20 17:58:59 WARN conf.Configuration: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
13/12/20 17:58:59 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
13/12/20 17:58:59 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
Metastore connection URL: jdbc:mysql://localhost:3306/hive_meta_store_mysql4_schematool?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: root
Starting upgrade metastore schema from version 0.9.0 to 0.12.0
Upgrade script upgrade-0.9.0-to-0.10.0.mysql.sql
Upgrade script upgrade-0.10.0-to-0.11.0.mysql.sql
Upgrade script upgrade-0.11.0-to-0.12.0.mysql.sql
schemaTool completeted
[cloud@Impetus-942 hadoop_migration]$ schematool -dbType mysql -upgradeSchemaFrom 0.9.0
13/12/20 17:59:19 WARN conf.Configuration: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
13/12/20 17:59:19 WARN conf.Configuration: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
13/12/20 17:59:19 WARN conf.Configuration: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
13/12/20 17:59:19 WARN conf.Configuration: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
13/12/20 17:59:19 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
13/12/20 17:59:19 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
Metastore connection URL: jdbc:mysql://localhost:3306/hive_meta_store_mysql4_schematool?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: root
Starting upgrade metastore schema from version 0.9.0 to 0.12.0
Upgrade script upgrade-0.9.0-to-0.10.0.mysql.sql
Completed upgrade-0.9.0-to-0.10.0.mysql.sql
Upgrade script upgrade-0.10.0-to-0.11.0.mysql.sql
Completed upgrade-0.10.0-to-0.11.0.mysql.sql
Upgrade script upgrade-0.11.0-to-0.12.0.mysql.sql
Completed upgrade-0.11.0-to-0.12.0.mysql.sql
schemaTool completeted
[cloud@Impetus-942 hadoop_migration]$
After this the following command gives me error:
[cloud@Impetus-942 hadoop_migration]$ schematool -dbType mysql -initSchema
13/12/20 18:00:59 WARN conf.Configuration: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
13/12/20 18:00:59 WARN conf.Configuration: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
13/12/20 18:00:59 WARN conf.Configuration: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
13/12/20 18:00:59 WARN conf.Configuration: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
13/12/20 18:00:59 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
13/12/20 18:00:59 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
Metastore connection URL: jdbc:mysql://localhost:3306/hive_meta_store_mysql4_schematool?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: root
Starting metastore schema initialization to 0.12.0
Initialization script hive-schema-0.12.0.mysql.sql
Error: Duplicate entry '1' for key 'PRIMARY' (state=23000,code=1062)
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
*** schemaTool failed ***
Can you please let me know what we mean by schematool -dbType mysql -initSchema ?
Also the *order* in which these commands needs to be invoked:
1. schematool -dbType mysql -initSchema
2. schematool -dbType mysql -upgradeSchemaFrom 0.9.0
Thanks,
-Nirmal
From: Lefty Leverenz [mailto:leftyleverenz@gmail.com<ma...@gmail.com>]
Sent: Friday, December 20, 2013 6:32 AM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
You should be aware of a new offline tool in Hive 0.12: The Hive Schema Tool<https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool#HiveSchemaTool-TheHiveSchemaTool>.
This tool can be used to initialize the metastore schema for the current Hive version. It can also handle upgrading the schema from an older version to current.
-- Lefty
On Wed, Dec 18, 2013 at 9:09 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Thanks Dima for pointing out the important info.
I'm doing a similar migration from *Hive 0.9 to Hive 0.12* and looking for some steps that needs to be performed as part of the migrate\upgrade.
Can you please share me the steps that you followed or referred?
Also I'm looking for what else needs to be considered if one goes for a migration of newer version of Hive from an older version.
I am assuming of some *backward compatibility* so that the existing queries that ran fine on the older version can run on the new version as well.
This way I assume there won't be any manual changes required in the queries.
I agree with you though that the queries needs to be tested thoroughly on the new version as well. Also yes there can be some new bugs with the new version.
-Nirmal
From: Dima Machlin [mailto:Dima.Machlin@pursway.com<ma...@pursway.com>]
Sent: Wednesday, December 18, 2013 4:17 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
Hi Nirmal,
I recently performed a similar upgrade process from Hive 0.7 to Hive 0.10
It seems that as some bugs in the 0.7 versions are solved in newer release, some new bugs appear.
I'd recommend to setup a test environment running the newer version of Hive, and testing the SQLs you are running on the current.
Then, compare the results in a one-to-one test checking that the data remains the same.
>From my experience I ran into 2 bugs in Hive 0.10 that don't appear in Hive 0.7 and required making some changes in the SQLs
The bugs are :
1) HIVE-5964<https://issues.apache.org/jira/browse/HIVE-5964> - a very specific edge case causing hive to miss a filter (thus causing wrong results)
2) HIVE-5607<https://issues.apache.org/jira/browse/HIVE-5607> - Hive failing to parse the "%" (mod) sign in certain conditions.
In your case you might come up with different bugs or none at all, but if your environment is a production environment I'd recommend performing a full test before upgrading.
Another option is to go over all the known bugs in v0.10 and see if they should affect your queries.
By doing so, you will be taking the risk of encountering unknown bugs.
From: Nirmal Kumar [mailto:nirmal.kumar@impetus.co.in]
Sent: Tuesday, December 17, 2013 8:49 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
Thanks Arun,
I will definitely move to the stable Hadoop-2.2.0 and Hive-0.12 versions.
All I want to know is whether upgrading\migrating from Hive 0.7/0.8/0.9 to Hive 0.12 requires some changes in the Hive queries or not? If yes where can I get the info?
-Nirmal
From: Arun C Murthy [mailto:acm@hortonworks.com]
Sent: Wednesday, December 18, 2013 12:07 AM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Upgrading Hive 0.7/0.8/0.9 to Hive 0.10.0
I'd strongly encourage you to go straight to hadoop-2.2.0 (the hadoop-2 GA release).
Also, I'd look at hive-0.12 which has been well tested against hadoop-2.2.0.
thanks,
Arun
On Dec 17, 2013, at 12:43 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I have migrated from Apache Hadoop 1.2.0 to Apache Hadoop 2.1.0-beta and now I want to migrate the Hive version(0.10.0 +) as well from an older version(0.7/0/8/0.9).
This is since the support for Hive to work on YARN in Hadoop 0.23 and 2.x releases has been supported since 0.10.0.
I wanted to know that if migrating from Hive 0.7/0.8/0.9 to Hive 0.10.0 requires some changes in the Hive queries or not. If the queries do need some changes then where can I get the info.?
Also are there some steps for migrating Hive 0.7/0.8/0.9 to Hive 0.10.0 ?
Thanks,
-Nirmal
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
************************************************************************************
This footnote confirms that this email message has been scanned by
PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses.
************************************************************************************
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.