You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/19 01:49:39 UTC
[GitHub] [spark] maropu commented on a change in pull request #32200: [SPARK-35102][SQL] Make spark.sql.hive.version read-only, not deprecated and meaningful
maropu commented on a change in pull request #32200:
URL: https://github.com/apache/spark/pull/32200#discussion_r615495829
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -53,22 +54,30 @@ private[spark] object HiveUtils extends Logging {
/** The version of hive used internally by Spark SQL. */
val builtinHiveVersion: String = HiveVersionInfo.getVersion
+ val BUILTIN_HIVE_VERSION = buildStaticConf("spark.sql.hive.version")
+ .doc(s"The compiled, a.k.a, builtin Hive version of the Spark distribution bundled with." +
+ s" Note that, this a read-only conf and only used to report the built-in hive version." +
+ s" If you want a different metastore client for Spark to call, please refer to" +
+ s" spark.sql.hive.metastore.version.")
Review comment:
nit: we can drop `s` at the beginning.
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -53,22 +54,30 @@ private[spark] object HiveUtils extends Logging {
/** The version of hive used internally by Spark SQL. */
val builtinHiveVersion: String = HiveVersionInfo.getVersion
+ val BUILTIN_HIVE_VERSION = buildStaticConf("spark.sql.hive.version")
+ .doc(s"The compiled, a.k.a, builtin Hive version of the Spark distribution bundled with." +
+ s" Note that, this a read-only conf and only used to report the built-in hive version." +
+ s" If you want a different metastore client for Spark to call, please refer to" +
+ s" spark.sql.hive.metastore.version.")
+ .version("1.1.1")
+ .stringConf
+ .checkValue(_ == builtinHiveVersion,
+ s"The builtin Hive version is read-only, please use spark.sql.hive.metastore.version")
+ .createWithDefault(builtinHiveVersion)
+
+ private def isCompatibleHiveVersion(hiveVersionStr: String): Boolean = {
+ Try { IsolatedClientLoader.hiveVersion(hiveVersionStr) }.isSuccess
+ }
+
val HIVE_METASTORE_VERSION = buildStaticConf("spark.sql.hive.metastore.version")
.doc("Version of the Hive metastore. Available options are " +
"<code>0.12.0</code> through <code>2.3.8</code> and " +
"<code>3.0.0</code> through <code>3.1.2</code>.")
.version("1.4.0")
.stringConf
+ .checkValue(ver => isCompatibleHiveVersion(ver), s"Unsupported Hive Metastore version")
.createWithDefault(builtinHiveVersion)
- // A fake config which is only here for backward compatibility reasons. This config has no effect
- // to Spark, just for reporting the builtin Hive version of Spark to existing applications that
- // already rely on this config.
Review comment:
We don't need to keep this comment above?
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -53,22 +54,30 @@ private[spark] object HiveUtils extends Logging {
/** The version of hive used internally by Spark SQL. */
val builtinHiveVersion: String = HiveVersionInfo.getVersion
+ val BUILTIN_HIVE_VERSION = buildStaticConf("spark.sql.hive.version")
+ .doc(s"The compiled, a.k.a, builtin Hive version of the Spark distribution bundled with." +
+ s" Note that, this a read-only conf and only used to report the built-in hive version." +
+ s" If you want a different metastore client for Spark to call, please refer to" +
+ s" spark.sql.hive.metastore.version.")
+ .version("1.1.1")
+ .stringConf
+ .checkValue(_ == builtinHiveVersion,
+ s"The builtin Hive version is read-only, please use spark.sql.hive.metastore.version")
+ .createWithDefault(builtinHiveVersion)
+
+ private def isCompatibleHiveVersion(hiveVersionStr: String): Boolean = {
+ Try { IsolatedClientLoader.hiveVersion(hiveVersionStr) }.isSuccess
+ }
+
val HIVE_METASTORE_VERSION = buildStaticConf("spark.sql.hive.metastore.version")
.doc("Version of the Hive metastore. Available options are " +
"<code>0.12.0</code> through <code>2.3.8</code> and " +
"<code>3.0.0</code> through <code>3.1.2</code>.")
.version("1.4.0")
.stringConf
+ .checkValue(ver => isCompatibleHiveVersion(ver), s"Unsupported Hive Metastore version")
Review comment:
`s"Unsupported Hive Metastore version: $ver"`?
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -53,22 +54,30 @@ private[spark] object HiveUtils extends Logging {
/** The version of hive used internally by Spark SQL. */
val builtinHiveVersion: String = HiveVersionInfo.getVersion
+ val BUILTIN_HIVE_VERSION = buildStaticConf("spark.sql.hive.version")
+ .doc(s"The compiled, a.k.a, builtin Hive version of the Spark distribution bundled with." +
+ s" Note that, this a read-only conf and only used to report the built-in hive version." +
+ s" If you want a different metastore client for Spark to call, please refer to" +
+ s" spark.sql.hive.metastore.version.")
+ .version("1.1.1")
+ .stringConf
+ .checkValue(_ == builtinHiveVersion,
+ s"The builtin Hive version is read-only, please use spark.sql.hive.metastore.version")
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org