You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Yongle Zhang (Jira)" <ji...@apache.org> on 2020/11/28 19:22:00 UTC
[jira] [Updated] (HIVE-24440) Upgrading Hive from 2.3.x(2.3.7) to
3.x(3.0.0), schema failed to upgrade
[ https://issues.apache.org/jira/browse/HIVE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yongle Zhang updated HIVE-24440:
--------------------------------
Description:
We tried to upgrade a hive single node from 2.3.7 to 3.0.0 and schema failed to upgrade because of a truncation error encountered when trying to shrink CHAR '2020-11-26 00:04:29.215' to length 1:
Error message:
{code:java}
0: jdbc:derby:> UPDATE "APP"."TBLS" SET "IS_REWRITE_ENABLED" = CURRENT_TIMESTAMP
Error: A truncation error was encountered trying to shrink CHAR '2020-11-26 00:04:29.215' to length 1. (state=22001,code=30000)
Closing: 0: jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:548)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1443)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1165)
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1143)
at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:543)
... 7 more
*** schemaTool failed ***
{code}
Environment:
Linux version 4.15.0-112-generic (buildd@lcy01-amd64-027) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
It can be reproduced through the following steps:
# Start up a standalone hive node of version 2.3.7 using embedded derby, and use schematool to initialize the schema.
# Use the following shell script to generate 20k data.
{code:java}
rm -rf ./data.txt
touch data.txt
for((i=0;i<20000;i++))
do
str=',name';
name=${i}${str}${i}
echo $name>> data.txt
done{code}
# Create a table and load data by using:
{code:java}
create table testTable(id int,name string) row format delimited fields terminated by ',';
LOAD DATA LOCAL INPATH '/data.txt' OVERWRITE INTO TABLE testTable;
{code}
# stop hive 2.3.7, and start hive compiled from 3.0.0, while keeping hadoop running.
# Run the following command to upgrade schema. Then we will see the previous error message.
{code:java}
schematool -dbType derby -upgradeSchema --verbose
{code}
# If we use the schematool to upgrade schema again, we will get the following error message:
{code:java}
0: jdbc:derby:> CREATE TABLE "APP"."METASTORE_DB_PROPERTIES" ("PROPERTY_KEY" VARCHAR(255) NOT NULL, "PROPERTY_VALUE" VARCHAR(1000) NOT NULL, "DESCRIPTION" VARCHAR(1000))
Error: Table/View 'METASTORE_DB_PROPERTIES' already exists in Schema 'APP'. (state=X0Y32,code=30000)
Closing: 0: jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:548)
at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:512)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1440)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1165)
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1143)
at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:543)
... 8 more
*** schemaTool failed ***
{code}
was:
We tried to upgrade a hive single node from 2.3.7 to 3.0.0 and schema failed to upgrade because of a truncation error encountered when trying to shrink CHAR '2020-11-26 00:04:29.215' to length 1:
Error message:
{code:java}
0: jdbc:derby:> UPDATE "APP"."TBLS" SET "IS_REWRITE_ENABLED" = CURRENT_TIMESTAMP
Error: A truncation error was encountered trying to shrink CHAR '2020-11-26 00:04:29.215' to length 1. (state=22001,code=30000)
Closing: 0: jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:548)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1443)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1165)
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1143)
at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:543)
... 7 more
*** schemaTool failed ***
{code}
It can be reproduced through the following steps:
# Start up a standalone hive node of version 2.3.7 using embedded derby, and use schematool to initialize the schema.
# Use the following shell script to generate 20k data.
{code:java}
rm -rf ./data.txt
touch data.txt
for((i=0;i<20000;i++))
do
str=',name';
name=${i}${str}${i}
echo $name>> data.txt
done{code}
# Create a table and load data by using:
{code:java}
create table testTable(id int,name string) row format delimited fields terminated by ',';
LOAD DATA LOCAL INPATH '/data.txt' OVERWRITE INTO TABLE testTable;
{code}
# stop hive 2.3.7, and start hive compiled from 3.0.0, while keeping hadoop running.
# Run the following command to upgrade schema. Then we will see the previous error message.
{code:java}
schematool -dbType derby -upgradeSchema --verbose
{code}
# If we use the schematool to upgrade schema again, we will get the following error message:
{code:java}
0: jdbc:derby:> CREATE TABLE "APP"."METASTORE_DB_PROPERTIES" ("PROPERTY_KEY" VARCHAR(255) NOT NULL, "PROPERTY_VALUE" VARCHAR(1000) NOT NULL, "DESCRIPTION" VARCHAR(1000))
Error: Table/View 'METASTORE_DB_PROPERTIES' already exists in Schema 'APP'. (state=X0Y32,code=30000)
Closing: 0: jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:548)
at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:512)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1440)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1165)
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1143)
at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:543)
... 8 more
*** schemaTool failed ***
{code}
> Upgrading Hive from 2.3.x(2.3.7) to 3.x(3.0.0), schema failed to upgrade
> -------------------------------------------------------------------------
>
> Key: HIVE-24440
> URL: https://issues.apache.org/jira/browse/HIVE-24440
> Project: Hive
> Issue Type: Bug
> Affects Versions: 2.3.7, 3.0.0
> Reporter: Yongle Zhang
> Priority: Major
>
> We tried to upgrade a hive single node from 2.3.7 to 3.0.0 and schema failed to upgrade because of a truncation error encountered when trying to shrink CHAR '2020-11-26 00:04:29.215' to length 1:
> Error message:
>
> {code:java}
> 0: jdbc:derby:> UPDATE "APP"."TBLS" SET "IS_REWRITE_ENABLED" = CURRENT_TIMESTAMP
> Error: A truncation error was encountered trying to shrink CHAR '2020-11-26 00:04:29.215' to length 1. (state=22001,code=30000)
> Closing: 0: jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
> at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:548)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1443)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1165)
> at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1143)
> at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:543)
> ... 7 more
> *** schemaTool failed ***
> {code}
>
> Environment:
> Linux version 4.15.0-112-generic (buildd@lcy01-amd64-027) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020
> openjdk version "1.8.0_252"
> OpenJDK Runtime Environment (build 1.8.0_252-b09)
> OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
> It can be reproduced through the following steps:
> # Start up a standalone hive node of version 2.3.7 using embedded derby, and use schematool to initialize the schema.
> # Use the following shell script to generate 20k data.
> {code:java}
> rm -rf ./data.txt
> touch data.txt
> for((i=0;i<20000;i++))
> do
> str=',name';
> name=${i}${str}${i}
> echo $name>> data.txt
> done{code}
> # Create a table and load data by using:
> {code:java}
> create table testTable(id int,name string) row format delimited fields terminated by ',';
> LOAD DATA LOCAL INPATH '/data.txt' OVERWRITE INTO TABLE testTable;
> {code}
> # stop hive 2.3.7, and start hive compiled from 3.0.0, while keeping hadoop running.
> # Run the following command to upgrade schema. Then we will see the previous error message.
> {code:java}
> schematool -dbType derby -upgradeSchema --verbose
> {code}
> # If we use the schematool to upgrade schema again, we will get the following error message:
> {code:java}
> 0: jdbc:derby:> CREATE TABLE "APP"."METASTORE_DB_PROPERTIES" ("PROPERTY_KEY" VARCHAR(255) NOT NULL, "PROPERTY_VALUE" VARCHAR(1000) NOT NULL, "DESCRIPTION" VARCHAR(1000))
> Error: Table/View 'METASTORE_DB_PROPERTIES' already exists in Schema 'APP'. (state=X0Y32,code=30000)
> Closing: 0: jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
> at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:548)
> at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:512)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1440)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1165)
> at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1143)
> at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:543)
> ... 8 more
> *** schemaTool failed ***
> {code}
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)