You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Alejandro Reina <al...@gmail.com> on 2017/12/10 11:17:44 UTC
Re: Save hive table from spark in hive 2.1.0
I have tried what you propose, added the property to hive-site.xml, and
although with this option I can run hive, this does not solve my problem.
I'm sorry if maybe you explain me wrongly.
I need to save a dataframe transformed into spark in hive, with the version
of scheme 2.1.1 of hive (last stable version of hive until 2 months ago), by
default spark internally uses version 1.2.1 and this generates conflict with
my other applications the ecosystem.
This hiveTable created from spark, is not visible from any other application
due to its version of the metadata, so I need to create from spark a hive
table in its corresponding version.
Hive schematool
root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -info --verbose
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:mysql://localhost/metastore?useSSL=false
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: hive
Hive distribution version: 2.1.0
Metastore schema version: 1.2.0
org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema version
is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0
org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema version
is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0
at
org.apache.hive.beeline.HiveSchemaTool.assertCompatibleVersion(HiveSchemaTool.java:202)
at org.apache.hive.beeline.HiveSchemaTool.showInfo(HiveSchemaTool.java:139)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:498)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
*** schemaTool failed ***
If i try upgrade schema too have error.
root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -upgradeSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:mysql://localhost/metastore?useSSL=false
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: hive
Starting upgrade metastore schema from version 1.2.0 to 2.1.0
Upgrade script upgrade-1.2.0-to-2.0.0.mysql.sql
Error: Duplicate column name 'CQ_HIGHEST_TXN_ID' (state=42S21,code=1060)
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED!
Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
Use --verbose for detailed stacktrace.
*** schemaTool failed ***
thank you very much for your time dedicated to helping me
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Save hive table from spark in hive 2.1.0
Posted by Alejandro Reina <al...@gmail.com>.
I did what you said and I was finally able to update the scheme. But you're
right, it's very dirty, I have to modify almost all the scripts.
The problem of the scripts comes from having already a previous table in
that version, many of the tables or columns that I try to add, already exist
and it gives many errors, but by modifying the script everything is ok.
A clearer way would be that Spark could save a hive table in the
corresponding version, but I do not know how to do it
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Save hive table from spark in hive 2.1.0
Posted by "☼ R Nair (रविशंकर नायर)" <ra...@gmail.com>.
Hi,
Good try. As you can see, when you run upgrade using schematool, there is a
duplicate column error. Can you please look the script generated and edit
to avoid duplicate column?
Not sure why the Hive guys made it complicated, I did face same issues like
you.
Can anyone else give a clean and better option?
Best, Ravion
On Dec 10, 2017 6:17 AM, "Alejandro Reina" <al...@gmail.com> wrote:
> I have tried what you propose, added the property to hive-site.xml, and
> although with this option I can run hive, this does not solve my problem.
> I'm sorry if maybe you explain me wrongly.
>
> I need to save a dataframe transformed into spark in hive, with the version
> of scheme 2.1.1 of hive (last stable version of hive until 2 months ago),
> by
> default spark internally uses version 1.2.1 and this generates conflict
> with
> my other applications the ecosystem.
>
> This hiveTable created from spark, is not visible from any other
> application
> due to its version of the metadata, so I need to create from spark a hive
> table in its corresponding version.
>
> Hive schematool
> root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -info --verbose
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-
> log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL: jdbc:mysql://localhost/
> metastore?useSSL=false
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> Hive distribution version: 2.1.0
> Metastore schema version: 1.2.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema
> version
> is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema
> version
> is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0
> at
> org.apache.hive.beeline.HiveSchemaTool.assertCompatibleVersion(
> HiveSchemaTool.java:202)
> at org.apache.hive.beeline.HiveSchemaTool.showInfo(
> HiveSchemaTool.java:139)
> at org.apache.hive.beeline.HiveSchemaTool.main(
> HiveSchemaTool.java:498)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> *** schemaTool failed ***
>
> If i try upgrade schema too have error.
> root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -upgradeSchema
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-
> log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL: jdbc:mysql://localhost/
> metastore?useSSL=false
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> Starting upgrade metastore schema from version 1.2.0 to 2.1.0
> Upgrade script upgrade-1.2.0-to-2.0.0.mysql.sql
> Error: Duplicate column name 'CQ_HIGHEST_TXN_ID' (state=42S21,code=1060)
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED!
> Metastore state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> Use --verbose for detailed stacktrace.
> *** schemaTool failed ***
>
> thank you very much for your time dedicated to helping me
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>