You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/02/02 09:55:28 UTC
[GitHub] [hudi] kingkongpoon opened a new issue #2517: [SUPPORT]Version information found in metastore differs
kingkongpoon opened a new issue #2517:
URL: https://github.com/apache/hudi/issues/2517
hive version 2.3.5
mysql's hive.VERSION SCHEMA_VERSION=2.3.0
When I successfully run my spark hudi job
```
8999 [main] INFO io.javalin.Javalin - Starting Javalin ...
9116 [main] INFO io.javalin.Javalin - Listening on http://localhost:40474/
9116 [main] INFO io.javalin.Javalin - Javalin started in 121ms \o/
15347 [main] INFO DataNucleus.Persistence - Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15347 [main] INFO DataNucleus.Persistence - Property datanucleus.cache.level2 unknown - will be ignored
17028 [main] INFO DataNucleus.Datastore - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
17029 [main] INFO DataNucleus.Datastore - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
17137 [main] INFO DataNucleus.Datastore - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
17137 [main] INFO DataNucleus.Datastore - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
17193 [main] INFO DataNucleus.Query - Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
17336 [main] ERROR org.apache.hadoop.hive.metastore.ObjectStore - Version information found in metastore differs 2.3.0 from expected schema version 1.2.0. Schema verififcation is disabled hive.metastore.schema.verification so setting version.
17641 [main] INFO DataNucleus.Datastore - The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
17683 [main] INFO hive.metastore - Mestastore configuration hive.metastore.warehouse.dir changed from /user/hive/warehouse to file:/root/spark-warehouse/
17690 [main] INFO DataNucleus.Query - Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
17948 [main] INFO io.javalin.Javalin - Stopping Javalin ...
17957 [main] INFO io.javalin.Javalin - Javalin has stopped
```
it throws an error ,17336 [main] ERROR
then I find it comes from
```
package org.apache.hadoop.hive.metastore
...
private synchronized void checkSchema() throws MetaException {
// recheck if it got verified by another thread while we were waiting
if (isSchemaVerified.get()) {
return;
}
boolean strictValidation =
HiveConf.getBoolVar(getConf(), HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION);
// read the schema version stored in metastore db
String dbSchemaVer = getMetaStoreSchemaVersion();
// version of schema for this version of hive
String hiveSchemaVer = MetaStoreSchemaInfo.getHiveSchemaVersion();
if (dbSchemaVer == null) {
if (strictValidation) {
throw new MetaException("Version information not found in metastore. ");
} else {
LOG.warn("Version information not found in metastore. "
+ HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() +
" is not enabled so recording the schema version " +
hiveSchemaVer);
setMetaStoreSchemaVersion(hiveSchemaVer,
"Set by MetaStore " + USER + "@" + HOSTNAME);
}
} else {
if (MetaStoreSchemaInfo.isVersionCompatible(hiveSchemaVer, dbSchemaVer)) {
LOG.debug("Found expected HMS version of " + dbSchemaVer);
} else {
// metastore schema version is different than Hive distribution needs
if (strictValidation) {
throw new MetaException("Hive Schema version " + hiveSchemaVer +
" does not match metastore's schema version " + dbSchemaVer +
" Metastore is not upgraded or corrupt");
} else {
LOG.error("Version information found in metastore differs " + dbSchemaVer +
" from expected schema version " + hiveSchemaVer +
". Schema verififcation is disabled " +
HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION);
setMetaStoreSchemaVersion(hiveSchemaVer,
"Set by MetaStore " + USER + "@" + HOSTNAME);
}
}
}
isSchemaVerified.set(true);
return;
}
...
```
Then the value (in mysql)hive.VERSION SCHEMA_VERSION=2.3.0 change to 1.2.0
And then I rerun my spark hudi jar ,it will not throw that error, but I can not restart my hive metastre ,because it's version change to 1.2.0 and hive --service metastore will throw error,
I put hive-site.xml in my maven project, include the mysql's user and password
I want to know why the spark hudi job can change the hive metadata which in mysql???
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2517: [SUPPORT]Version information found in metastore differs
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2517:
URL: https://github.com/apache/hudi/issues/2517#issuecomment-774142928
@n3nash : I don't hv much exp w/ hive. Can you take this up or add someone who can help here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #2517: [SUPPORT]Version information found in metastore differs
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #2517:
URL: https://github.com/apache/hudi/issues/2517#issuecomment-855195036
Can you try with a higher hive version or reproduce this in the docker setup? We run an integ test on each commit, that works on Apache Hive 2.3.1. So I am a bit confused as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #2517: [SUPPORT]Version information found in metastore differs
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #2517:
URL: https://github.com/apache/hudi/issues/2517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2517: [SUPPORT]Version information found in metastore differs
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2517:
URL: https://github.com/apache/hudi/issues/2517#issuecomment-908907330
This does not seem like an issue as such, since our CI tests w/ apache hive for every commit. Closing it out. Probably some environment issue in relation to hive-site.xml. Closing it for now. Please re-open if you need more help in this regard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #2517: [SUPPORT]Version information found in metastore differs
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #2517:
URL: https://github.com/apache/hudi/issues/2517#issuecomment-774419451
Are you trying to make Hudi working Hive 1.x? Or the 1.2 is the Cloudera version? Can you give us more context on the different hive versions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2517: [SUPPORT]Version information found in metastore differs
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2517:
URL: https://github.com/apache/hudi/issues/2517#issuecomment-908907330
This does not seem like an issue as such, since our CI tests w/ apache hive for every commit. Closing it out. Probably some environment issue in relation to hive-site.xml. Closing it for now. Please re-open if you need more help in this regard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #2517: [SUPPORT]Version information found in metastore differs
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #2517:
URL: https://github.com/apache/hudi/issues/2517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kingkongpoon commented on issue #2517: [SUPPORT]Version information found in metastore differs
Posted by GitBox <gi...@apache.org>.
kingkongpoon commented on issue #2517:
URL: https://github.com/apache/hudi/issues/2517#issuecomment-774576921
My hive version is apache-hive-2.3.5，the apache official version，when I us alibaba's EMR(hive version is also 2.3.5) , it will not happend.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org