You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/09/14 09:55:38 UTC

(spark) branch branch-4.x updated: [SPARK-59330][SQL] Add an option to defer Hive InputFormat and OutputFormat class initialization

This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new ba87e94537a9 [SPARK-59330][SQL] Add an option to defer Hive InputFormat and OutputFormat class initialization
ba87e94537a9 is described below

commit ba87e94537a90a52cbc9b61fcabbb68017f987b3
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Mon Sep 14 18:55:09 2026 +0900

    [SPARK-59330][SQL] Add an option to defer Hive InputFormat and OutputFormat class initialization
    
    ### What changes were proposed in this pull request?
    
    When Spark converts Hive metastore metadata into a Hive table (`HiveClientImpl.toHiveTable`, on the
    driver), it resolves the table's `InputFormat`/`OutputFormat` class names. Resolving a class with
    `Class.forName(..., initialize = true)` (the default) runs the class's static initializer, even
    though only the class name is needed here (Hive stores it via `getName`).
    
    This adds `spark.sql.hive.initializeMetastoreFormatClasses` (default `true`, the previous
    behavior). When set to `false`, the format class is still loaded (so a missing class still fails
    here), but its static initializer is not run at resolution time; it runs later when the format is
    actually instantiated for a scan/write.
    
    ### Why are the changes needed?
    
    Running a format class's static initializer merely to read its name during a metadata operation is
    unnecessary. The option lets an operator resolve these class names without triggering static
    initialization at metadata-conversion time.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No by default: `spark.sql.hive.initializeMetastoreFormatClasses` defaults to `true`, preserving the
    previous behavior. When set to `false`, a format class's static initializer runs when the format is
    instantiated rather than when the metastore metadata is resolved. For a class whose static
    initializer fails, this changes when the failure surfaces: no longer fast at resolution time, but
    when the format is instantiated -- which for a Hive serde scan is on the driver (when the input
    splits are computed) and for a Hive serde write is on an executor. A table read through the
    built-in Parquet/ORC reader never instantiates the format, so there the initializer never runs.
    
    ### How was this patch tested?
    
    New `HiveClientImplSuite` tests:
    - `SPARK-59330: toHiveTable skips the format class static initializer when
      spark.sql.hive.initializeMetastoreFormatClasses is false`: with the option `false`, resolving a
      format class name via `toHiveTable` does not run its static initializer, and with the option
      `true` (the default) it does. It covers both the `toInputFormat` and `toOutputFormat` call sites,
      using two new test-only format classes (`StaticInitInputFormat`, `StaticInitOutputFormat`) whose
      static initializers flip flags on a separate `StaticInitFlags` holder (read without initializing
      the format classes).
    - `SPARK-59330: toHiveTable still resolves the format class when
      spark.sql.hive.initializeMetastoreFormatClasses is false`: with the option `false`, a nonexistent
      format class still throws `ClassNotFoundException` at resolution time, pinning that the class is
      still loaded (the reason for `initialize = false` over dropping the resolution entirely).
    
    The default path continues to be covered by the existing Hive tests.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 4.8
    
    This pull request and its description were written by Isaac.
    
    Closes #58615 from HyukjinKwon/SPARK-59330-hive-inputformat-no-init.
    
    Authored-by: Hyukjin Kwon <gu...@apache.org>
    Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
    (cherry picked from commit b87b39c35699979b5a1e99805aa7dc1b03bfcfb4)
    Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
---
 .../org/apache/spark/sql/hive/HiveUtils.scala      | 20 +++++++
 .../spark/sql/hive/client/HiveClientImpl.scala     | 16 +++++-
 .../org/apache/spark/sql/hive/StaticInitFlags.java | 31 +++++++++++
 .../spark/sql/hive/StaticInitInputFormat.java      | 46 ++++++++++++++++
 .../spark/sql/hive/StaticInitOutputFormat.java     | 62 ++++++++++++++++++++++
 .../sql/hive/client/HiveClientImplSuite.scala      | 57 ++++++++++++++++++++
 6 files changed, 230 insertions(+), 2 deletions(-)

diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
index 4028da153ff9..8ef6e0cca538 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
@@ -199,6 +199,26 @@ private[spark] object HiveUtils extends Logging {
     .booleanConf
     .createWithDefault(false)
 
+  val INITIALIZE_METASTORE_FORMAT_CLASSES =
+    buildConf("spark.sql.hive.initializeMetastoreFormatClasses")
+      .doc("When true, an InputFormat/OutputFormat class name stored in the Hive metastore is " +
+        "resolved with its static initializer run at resolution time. When false, the class is " +
+        "still loaded, so a missing class still fails here, but its static initializer is not " +
+        "run. Only the class name is needed when converting metastore metadata, so setting " +
+        "this to false avoids running a format class's static initializer during a metadata " +
+        "operation. Note that the conversion also runs when planning a scan or write and when " +
+        "inferring the schema of a Hive serde table, so with false a format class whose static " +
+        "initializer fails no longer fails fast at resolution time. It fails when the format " +
+        "is instantiated instead, " +
+        "which for a Hive serde scan is on the driver when the input splits are computed, and " +
+        "for a Hive serde write is on an executor. A table read through the built-in " +
+        "Parquet/ORC reader never instantiates the format, so there the initializer never " +
+        "runs. Keep this true (the default) to preserve the fail-fast behavior.")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .booleanConf
+      .createWithDefault(true)
+
   val HIVE_METASTORE_SHARED_PREFIXES = buildStaticConf("spark.sql.hive.metastore.sharedPrefixes")
     .doc("A comma separated list of class prefixes that should be loaded using the classloader " +
       "that is shared between Spark SQL and a specific version of Hive. An example of classes " +
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index f09f076707be..6c265ce6a43f 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -1158,11 +1158,23 @@ private[hive] object HiveClientImpl extends Logging {
     Option(hc.getComment).map(field.withComment).getOrElse(field)
   }
 
+  // Converting metastore metadata does not need the format class initialized: the resolved
+  // Class is only stored (via setInputFormatClass/setOutputFormatClass, which read getName),
+  // compared by identity, or tested with isAssignableFrom downstream, and it is instantiated
+  // through ReflectionUtils, which initializes it there. So when
+  // spark.sql.hive.initializeMetastoreFormatClasses is false the class is resolved without
+  // running its static initializer here; the initializer then runs when the format is actually
+  // instantiated for a scan/write. Defaults to true (the previous behavior).
+  private def initializeFormatClasses: Boolean =
+    SQLConf.get.getConf(HiveUtils.INITIALIZE_METASTORE_FORMAT_CLASSES)
+
   private def toInputFormat(name: String) =
-    Utils.classForName[org.apache.hadoop.mapred.InputFormat[_, _]](name)
+    Utils.classForName[org.apache.hadoop.mapred.InputFormat[_, _]](
+      name, initialize = initializeFormatClasses)
 
   private def toOutputFormat(name: String) =
-    Utils.classForName[org.apache.hadoop.hive.ql.io.HiveOutputFormat[_, _]](name)
+    Utils.classForName[org.apache.hadoop.hive.ql.io.HiveOutputFormat[_, _]](
+      name, initialize = initializeFormatClasses)
 
   def toHiveTableType(catalogTableType: CatalogTableType): HiveTableType = {
     catalogTableType match {
diff --git a/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitFlags.java b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitFlags.java
new file mode 100644
index 000000000000..5a8cc2f02e3b
--- /dev/null
+++ b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitFlags.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive;
+
+/**
+ * Test-only holder for flags flipped by the static initializers of {@link StaticInitInputFormat}
+ * and {@link StaticInitOutputFormat}. The flags live on a separate class on purpose: a test can
+ * read them to observe whether a format class was initialized, without initializing the format
+ * class itself (which reading a field off it would do).
+ */
+public final class StaticInitFlags {
+  public static volatile boolean inputFormatInitialized = false;
+  public static volatile boolean outputFormatInitialized = false;
+
+  private StaticInitFlags() {}
+}
diff --git a/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitInputFormat.java b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitInputFormat.java
new file mode 100644
index 000000000000..59564967fd39
--- /dev/null
+++ b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitInputFormat.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive;
+
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hadoop.mapred.Reporter;
+
+/**
+ * Test-only {@link InputFormat} whose static initializer flips
+ * {@link StaticInitFlags#inputFormatInitialized}. It lets a test verify whether resolving the
+ * format class by name runs its static initializer. The reader methods are never called.
+ */
+public class StaticInitInputFormat implements InputFormat<Void, Void> {
+  static {
+    StaticInitFlags.inputFormatInitialized = true;
+  }
+
+  @Override
+  public InputSplit[] getSplits(JobConf job, int numSplits) {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public RecordReader<Void, Void> getRecordReader(
+      InputSplit split, JobConf job, Reporter reporter) {
+    throw new UnsupportedOperationException();
+  }
+}
diff --git a/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitOutputFormat.java b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitOutputFormat.java
new file mode 100644
index 000000000000..ab40c9247d6c
--- /dev/null
+++ b/sql/hive/src/test/java/org/apache/spark/sql/hive/StaticInitOutputFormat.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive;
+
+import java.util.Properties;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordWriter;
+import org.apache.hadoop.util.Progressable;
+
+/**
+ * Test-only {@link HiveOutputFormat} whose static initializer flips
+ * {@link StaticInitFlags#outputFormatInitialized}. It lets a test verify whether resolving the
+ * format class by name runs its static initializer. The writer methods are never called.
+ */
+public class StaticInitOutputFormat implements HiveOutputFormat<Void, Void> {
+  static {
+    StaticInitFlags.outputFormatInitialized = true;
+  }
+
+  @Override
+  public FileSinkOperator.RecordWriter getHiveRecordWriter(
+      JobConf jc,
+      Path finalOutPath,
+      Class<? extends Writable> valueClass,
+      boolean isCompressed,
+      Properties tableProperties,
+      Progressable progress) {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public RecordWriter<Void, Void> getRecordWriter(
+      FileSystem ignored, JobConf job, String name, Progressable progress) {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public void checkOutputSpecs(FileSystem ignored, JobConf job) {
+    throw new UnsupportedOperationException();
+  }
+}
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientImplSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientImplSuite.scala
index 9bee44f8f57e..6dd3b16637b3 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientImplSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientImplSuite.scala
@@ -20,9 +20,66 @@ package org.apache.spark.sql.hive.client
 import org.apache.hadoop.hive.metastore.api.FieldSchema
 
 import org.apache.spark.{SparkFunSuite, SparkUnsupportedOperationException}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.hive.{HiveUtils, StaticInitFlags, StaticInitInputFormat, StaticInitOutputFormat}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
 
 class HiveClientImplSuite extends SparkFunSuite {
 
+  test("SPARK-59330: toHiveTable skips the format class static initializer when " +
+    "spark.sql.hive.initializeMetastoreFormatClasses is false") {
+    // Both call sites are exercised: toHiveTable resolves the input format via toInputFormat and
+    // the output format via toOutputFormat, so each format class has its own flag.
+    val table = CatalogTable(
+      identifier = TableIdentifier("t", Some("default")),
+      tableType = CatalogTableType.MANAGED,
+      storage = CatalogStorageFormat.empty.copy(
+        inputFormat = Some(classOf[StaticInitInputFormat].getName),
+        outputFormat = Some(classOf[StaticInitOutputFormat].getName)),
+      schema = new StructType().add("a", "int"))
+
+    def toHiveTableWith(initialize: Boolean): Unit = {
+      val conf = new SQLConf()
+      conf.setConf(HiveUtils.INITIALIZE_METASTORE_FORMAT_CLASSES, initialize)
+      SQLConf.withExistingConf(conf) {
+        HiveClientImpl.toHiveTable(table)
+      }
+    }
+
+    // The false half must run first: class initialization is one-way per JVM. Resolving the
+    // format class names without initializing them must not run their static initializers.
+    toHiveTableWith(initialize = false)
+    assert(!StaticInitFlags.inputFormatInitialized)
+    assert(!StaticInitFlags.outputFormatInitialized)
+
+    // With initialization enabled (the default), resolving the class names runs the initializers.
+    toHiveTableWith(initialize = true)
+    assert(StaticInitFlags.inputFormatInitialized)
+    assert(StaticInitFlags.outputFormatInitialized)
+  }
+
+  test("SPARK-59330: toHiveTable still resolves the format class when " +
+    "spark.sql.hive.initializeMetastoreFormatClasses is false") {
+    // false skips the static initializer but still loads the class, so a missing class fails here
+    // rather than being silently deferred; this pins that contract (the reason for initialize =
+    // false over dropping the resolution entirely).
+    val table = CatalogTable(
+      identifier = TableIdentifier("t", Some("default")),
+      tableType = CatalogTableType.MANAGED,
+      storage = CatalogStorageFormat.empty.copy(
+        inputFormat = Some("org.apache.spark.sql.hive.DoesNotExistInputFormat")),
+      schema = new StructType().add("a", "int"))
+    val conf = new SQLConf()
+    conf.setConf(HiveUtils.INITIALIZE_METASTORE_FORMAT_CLASSES, false)
+    intercept[ClassNotFoundException] {
+      SQLConf.withExistingConf(conf) {
+        HiveClientImpl.toHiveTable(table)
+      }
+    }
+  }
+
   test("SPARK-21529: a clear error is raised for an unsupported Hive union type") {
     val column = new FieldSchema("c", "uniontype<int,string>", null)
     checkError(


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org