You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/09/14 09:55:38 UTC
(spark) branch branch-4.x updated: [SPARK-59330][SQL] Add an option to defer Hive InputFormat and OutputFormat class initialization
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ba87e94537a9 [SPARK-59330][SQL] Add an option to defer Hive InputFormat and OutputFormat class initialization
ba87e94537a9 is described below
commit ba87e94537a90a52cbc9b61fcabbb68017f987b3
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Mon Sep 14 18:55:09 2026 +0900
[SPARK-59330][SQL] Add an option to defer Hive InputFormat and OutputFormat class initialization
### What changes were proposed in this pull request?
When Spark converts Hive metastore metadata into a Hive table (`HiveClientImpl.toHiveTable`, on the
driver), it resolves the table's `InputFormat`/`OutputFormat` class names. Resolving a class with
`Class.forName(..., initialize = true)` (the default) runs the class's static initializer, even
though only the class name is needed here (Hive stores it via `getName`).
This adds `spark.sql.hive.initializeMetastoreFormatClasses` (default `true`, the previous
behavior). When set to `false`, the format class is still loaded (so a missing class still fails
here), but its static initializer is not run at resolution time; it runs later when the format is
actually instantiated for a scan/write.
### Why are the changes needed?
Running a format class's static initializer merely to read its name during a metadata operation is
unnecessary. The option lets an operator resolve these class names without triggering static
initialization at metadata-conversion time.
### Does this PR introduce _any_ user-facing change?
No by default: `spark.sql.hive.initializeMetastoreFormatClasses` defaults to `true`, preserving the
previous behavior. When set to `false`, a format class's static initializer runs when the format is
instantiated rather than when the metastore metadata is resolved. For a class whose static
initializer fails, this changes when the failure surfaces: no longer fast at resolution time, but
when the format is instantiated -- which for a Hive serde scan is on the driver (when the input
splits are computed) and for a Hive serde write is on an executor. A table read through the
built-in Parquet/ORC reader never instantiates the format, so there the initializer never runs.
### How was this patch tested?
New `HiveClientImplSuite` tests:
- `SPARK-59330: toHiveTable skips the format class static initializer when
spark.sql.hive.initializeMetastoreFormatClasses is false`: with the option `false`, resolving a
format class name via `toHiveTable` does not run its static initializer, and with the option
`true` (the default) it does. It covers both the `toInputFormat` and `toOutputFormat` call sites,
using two new test-only format classes (`StaticInitInputFormat`, `StaticInitOutputFormat`) whose
static initializers flip flags on a separate `StaticInitFlags` holder (read without initializing
the format classes).
- `SPARK-59330: toHiveTable still resolves the format class when
spark.sql.hive.initializeMetastoreFormatClasses is false`: with the option `false`, a nonexistent
format class still throws `ClassNotFoundException` at resolution time, pinning that the class is
still loaded (the reason for `initialize = false` over dropping the resolution entirely).
The default path continues to be covered by the existing Hive tests.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
This pull request and its description were written by Isaac.
Closes #58615 from HyukjinKwon/SPARK-59330-hive-inputformat-no-init.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
(cherry picked from commit b87b39c35699979b5a1e99805aa7dc1b03bfcfb4)
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
---
.../org/apache/spark/sql/hive/HiveUtils.scala | 20 +++++++
.../spark/sql/hive/client/HiveClientImpl.scala | 16 +++++-
.../org/apache/spark/sql/hive/StaticInitFlags.java | 31 +++++++++++
.../spark/sql/hive/StaticInitInputFormat.java | 46 ++++++++++++++++
.../spark/sql/hive/StaticInitOutputFormat.java | 62 ++++++++++++++++++++++
.../sql/hive/client/HiveClientImplSuite.scala | 57 ++++++++++++++++++++
6 files changed, 230 insertions(+), 2 deletions(-)
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
index 4028da153ff9..8ef6e0cca538 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
@@ -199,6 +199,26 @@ private[spark] object HiveUtils extends Logging {
.booleanConf
.createWithDefault(false)
+ val INITIALIZE_METASTORE_FORMAT_CLASSES =
+ buildConf("spark.sql.hive.initializeMetastoreFormatClasses")
+ .doc("When true, an InputFormat/OutputFormat class name stored in the Hive metastore is " +
+ "resolved with its static initializer run at resolution time. When false, the class is " +
+ "still loaded, so a missing class still fails here, but its static initializer is not " +
+ "run. Only the class name is needed when converting metastore metadata, so setting " +
+ "this to false avoids running a format class's static initializer during a metadata " +
+ "operation. Note that the conversion also runs when planning a scan or write and when " +
+ "inferring the schema of a Hive serde table, so with false a format class whose static " +
+ "initializer fails no longer fails fast at resolution time. It fails when the format " +
+ "is instantiated instead, " +
+ "which for a Hive serde scan is on the driver when the input splits are computed, and " +
+ "for a Hive serde write is on an executor. A table read through the built-in " +
+ "Parquet/ORC reader never instantiates the format, so there the initializer never " +
+ "runs. Keep this true (the default) to preserve the fail-fast behavior.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(true)
+
val HIVE_METASTORE_SHARED_PREFIXES = buildStaticConf("spark.sql.hive.metastore.sharedPrefixes")
.doc("A comma separated list of class prefixes that should be loaded using the classloader " +
"that is shared between Spark SQL and a specific version of Hive. An example of classes " +
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index f09f076707be..6c265ce6a43f 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -1158,11 +1158,23 @@ private[hive] object HiveClientImpl extends Logging {
Option(hc.getComment).map(field.withComment).getOrElse(field)
}
+ // Converting metastore metadata does not need the format class initialized: the resolved
+ // Class is only stored (via setInputFormatClass/setOutputFormatClass, which read getName),
+ // compared by identity, or tested with isAssignableFrom downstream, and it is instantiated
+ // through ReflectionUtils, which initializes it there. So when
+ // spark.sql.hive.initializeMetastoreFormatClasses is false the class is resolved without
+ // running its static initializer here; the initializer then runs when the format is actually
+ // instantiated for a scan/write. Defaults to true (the previous behavior).
+ private def initializeFormatClasses: Boolean =
+ SQLConf.get.getConf(HiveUtils.INITIALIZE_METASTORE_FORMAT_CLASSES)
+
private def toInputFormat(name: String) =
- Utils.classForName[org.apache.hadoop.mapred.InputFormat[_, _]](name)
+ Utils.classForName[org.apache.hadoop.mapred.InputFormat[_, _]](
+ name, initialize = initializeFormatClasses)
private def toOutputFormat(name: String) =
- Utils.classForName[org.apache.hadoop.hive.ql.io.HiveOutputFormat[_, _]](name)
+ Utils.classForName[org.apache.hadoop.hive.ql.io.HiveOutputFormat[_, _]](
+ name, initialize = initializeFormatClasses)
def toHiveTableType(catalogTableType: CatalogTableType): HiveTableType = {
catalogTableType match {
diff --git a/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitFlags.java b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitFlags.java
new file mode 100644
index 000000000000..5a8cc2f02e3b
--- /dev/null
+++ b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitFlags.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive;
+
+/**
+ * Test-only holder for flags flipped by the static initializers of {@link StaticInitInputFormat}
+ * and {@link StaticInitOutputFormat}. The flags live on a separate class on purpose: a test can
+ * read them to observe whether a format class was initialized, without initializing the format
+ * class itself (which reading a field off it would do).
+ */
+public final class StaticInitFlags {
+ public static volatile boolean inputFormatInitialized = false;
+ public static volatile boolean outputFormatInitialized = false;
+
+ private StaticInitFlags() {}
+}
diff --git a/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitInputFormat.java b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitInputFormat.java
new file mode 100644
index 000000000000..59564967fd39
--- /dev/null
+++ b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitInputFormat.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive;
+
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hadoop.mapred.Reporter;
+
+/**
+ * Test-only {@link InputFormat} whose static initializer flips
+ * {@link StaticInitFlags#inputFormatInitialized}. It lets a test verify whether resolving the
+ * format class by name runs its static initializer. The reader methods are never called.
+ */
+public class StaticInitInputFormat implements InputFormat<Void, Void> {
+ static {
+ StaticInitFlags.inputFormatInitialized = true;
+ }
+
+ @Override
+ public InputSplit[] getSplits(JobConf job, int numSplits) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public RecordReader<Void, Void> getRecordReader(
+ InputSplit split, JobConf job, Reporter reporter) {
+ throw new UnsupportedOperationException();
+ }
+}
diff --git a/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitOutputFormat.java b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitOutputFormat.java
new file mode 100644
index 000000000000..ab40c9247d6c
--- /dev/null
+++ b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitOutputFormat.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive;
+
+import java.util.Properties;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordWriter;
+import org.apache.hadoop.util.Progressable;
+
+/**
+ * Test-only {@link HiveOutputFormat} whose static initializer flips
+ * {@link StaticInitFlags#outputFormatInitialized}. It lets a test verify whether resolving the
+ * format class by name runs its static initializer. The writer methods are never called.
+ */
+public class StaticInitOutputFormat implements HiveOutputFormat<Void, Void> {
+ static {
+ StaticInitFlags.outputFormatInitialized = true;
+ }
+
+ @Override
+ public FileSinkOperator.RecordWriter getHiveRecordWriter(
+ JobConf jc,
+ Path finalOutPath,
+ Class<? extends Writable> valueClass,
+ boolean isCompressed,
+ Properties tableProperties,
+ Progressable progress) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public RecordWriter<Void, Void> getRecordWriter(
+ FileSystem ignored, JobConf job, String name, Progressable progress) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void checkOutputSpecs(FileSystem ignored, JobConf job) {
+ throw new UnsupportedOperationException();
+ }
+}
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientImplSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientImplSuite.scala
index 9bee44f8f57e..6dd3b16637b3 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientImplSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientImplSuite.scala
@@ -20,9 +20,66 @@ package org.apache.spark.sql.hive.client
import org.apache.hadoop.hive.metastore.api.FieldSchema
import org.apache.spark.{SparkFunSuite, SparkUnsupportedOperationException}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.hive.{HiveUtils, StaticInitFlags, StaticInitInputFormat, StaticInitOutputFormat}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
class HiveClientImplSuite extends SparkFunSuite {
+ test("SPARK-59330: toHiveTable skips the format class static initializer when " +
+ "spark.sql.hive.initializeMetastoreFormatClasses is false") {
+ // Both call sites are exercised: toHiveTable resolves the input format via toInputFormat and
+ // the output format via toOutputFormat, so each format class has its own flag.
+ val table = CatalogTable(
+ identifier = TableIdentifier("t", Some("default")),
+ tableType = CatalogTableType.MANAGED,
+ storage = CatalogStorageFormat.empty.copy(
+ inputFormat = Some(classOf[StaticInitInputFormat].getName),
+ outputFormat = Some(classOf[StaticInitOutputFormat].getName)),
+ schema = new StructType().add("a", "int"))
+
+ def toHiveTableWith(initialize: Boolean): Unit = {
+ val conf = new SQLConf()
+ conf.setConf(HiveUtils.INITIALIZE_METASTORE_FORMAT_CLASSES, initialize)
+ SQLConf.withExistingConf(conf) {
+ HiveClientImpl.toHiveTable(table)
+ }
+ }
+
+ // The false half must run first: class initialization is one-way per JVM. Resolving the
+ // format class names without initializing them must not run their static initializers.
+ toHiveTableWith(initialize = false)
+ assert(!StaticInitFlags.inputFormatInitialized)
+ assert(!StaticInitFlags.outputFormatInitialized)
+
+ // With initialization enabled (the default), resolving the class names runs the initializers.
+ toHiveTableWith(initialize = true)
+ assert(StaticInitFlags.inputFormatInitialized)
+ assert(StaticInitFlags.outputFormatInitialized)
+ }
+
+ test("SPARK-59330: toHiveTable still resolves the format class when " +
+ "spark.sql.hive.initializeMetastoreFormatClasses is false") {
+ // false skips the static initializer but still loads the class, so a missing class fails here
+ // rather than being silently deferred; this pins that contract (the reason for initialize =
+ // false over dropping the resolution entirely).
+ val table = CatalogTable(
+ identifier = TableIdentifier("t", Some("default")),
+ tableType = CatalogTableType.MANAGED,
+ storage = CatalogStorageFormat.empty.copy(
+ inputFormat = Some("org.apache.spark.sql.hive.DoesNotExistInputFormat")),
+ schema = new StructType().add("a", "int"))
+ val conf = new SQLConf()
+ conf.setConf(HiveUtils.INITIALIZE_METASTORE_FORMAT_CLASSES, false)
+ intercept[ClassNotFoundException] {
+ SQLConf.withExistingConf(conf) {
+ HiveClientImpl.toHiveTable(table)
+ }
+ }
+ }
+
test("SPARK-21529: a clear error is raised for an unsupported Hive union type") {
val column = new FieldSchema("c", "uniontype<int,string>", null)
checkError(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org