You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pt...@apache.org on 2026/10/16 15:43:22 UTC

(spark) branch branch-4.x updated: [SPARK-60262][SQL] Fail Parquet reads where two requested fields select the same column

This is an automated email from the ASF dual-hosted git repository.

peter-toth pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 4af89ec7760c [SPARK-60262][SQL] Fail Parquet reads where two requested fields select the same column
4af89ec7760c is described below

commit 4af89ec7760ceefc664fd6dc425bafce98258e1c
Author: Andy Grove <ag...@apache.org>
AuthorDate: Fri Oct 16 17:42:54 2026 +0200

    [SPARK-60262][SQL] Fail Parquet reads where two requested fields select the same column
    
    ### What changes were proposed in this pull request?
    
    With `spark.sql.parquet.fieldId.read.enabled=true`, `ParquetReadSupport.clipParquetGroupFields` resolves each requested field on its own: a field with a field id by id, and any other field by name. Two requested fields can therefore be clipped to the same Parquet field, and neither reader can read a column twice (see below). This PR makes such a read fail with a clear error instead of returning wrong values:
    
    - `clipParquetGroupFields` throws the new `PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE` error when two or more requested fields of a struct are clipped to the same Parquet field. That covers a field id match plus a name match, and two fields with the same id, at the top level and inside nested structs. The check is in the clipping code that both readers use, so it covers the vectorized reader and the parquet-mr reader.
    - `VectorizedColumnReader.readBatch` throws a `ParquetDecodingException` when a column without repeated values runs out of pages with rows still to read, instead of returning. With the check above nothing in Spark reaches this, but returning leaves the rest of the vector unwritten without any signal, so this keeps a column that is ever read twice, or a truncated column chunk, from quietly returning wrong values. Repeated columns still stop at the end of the pages, because the end of a [...]
    
    ### Why are the changes needed?
    
    Take a file column `A` with field id 1. A requested `x` with id 1 selects it by id, and a requested `a` without an id selects it by name, so the requested Parquet schema lists `A` twice. Neither reader returns `A`'s values for both fields, and neither reports an error:
    
    - The vectorized reader builds one `VectorizedColumnReader` per requested leaf, but a Parquet page store has a single queue of pages per column. The first reader drains it. For the second, `readPage()` returns -1 and `readBatch` stops as if it had read every page, so its vector is never written, and its rows read as non-null `0` with on-heap vectors, or as whatever the memory holds with off-heap vectors.
    - The parquet-mr reader (`spark.sql.parquet.enableVectorizedReader=false`) has `ParquetRowConverter` map the clipped Parquet fields back to the requested fields by id first. Both copies of `A` carry id 1, so both map to `x`, and `a` reads null.
    
    ```scala
    val id1 = new MetadataBuilder().putLong("parquet.field.id", 1L).build()
    spark.createDataFrame(
        spark.sparkContext.parallelize(Seq(Row(1), Row(2), Row(3)), 1),
        StructType(Seq(StructField("A", IntegerType, nullable = true, id1))))
      .write.parquet("/tmp/field-id-twice")
    
    spark.conf.set("spark.sql.parquet.fieldId.read.enabled", "true")
    val x = StructField("x", IntegerType, nullable = true, id1)
    val a = StructField("a", IntegerType, nullable = true)
    spark.read.schema(StructType(Seq(x, a))).parquet("/tmp/field-id-twice").show()
    ```
    
    Before this change, `x` reads 1, 2, 3 and `a` reads 0, 0, 0 with the vectorized reader (the other way round when `a` comes first), and `a` reads null with the parquet-mr reader. A struct holding `x` and `a` behaves the same.
    
    Reading both fields correctly would take changes to how both readers map clipped fields back to the requested ones, and a way for the vectorized reader to serve one column to several leaves. This PR is the smaller step of replacing the silent wrong results with an error. Field-id reads (SPARK-38094) have returned these results since 3.3.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Reading a schema in which two requested fields select the same Parquet column, which returned wrong values without an error (see above), now fails with `PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE`. The wrong results come from field-id reads, so released versions since 3.3 have them as well. Reads that do not select a column more than once are not affected.
    
    ### How was this patch tested?
    
    New tests:
    - `ParquetFieldIdSchemaSuite`: `ParquetReadSupport.clipParquetSchema` fails, naming the requested fields and the Parquet field, for an id match plus a name match in either order (case-sensitive and not), two fields with the same id, a third field matching by name, and a nested struct. It does not fail when the fields select different Parquet fields, or none.
    - `ParquetFieldIdIOSuite`: reads with the vectorized and the parquet-mr reader fail for the same shapes (top level, nested, same id), and a requested field that matches no column still reads as null.
    - `ParquetVectorizedSuite`: a requested schema that lists a column twice, over a page store that shares one page reader per column as Parquet's does, fails in `VectorizedColumnReader.readBatch` instead of leaving the second vector unwritten.
    
    The new tests fail without this change (no exception is thrown, and the data reads back wrong) and pass with it. The `ParquetVectorizedSuite` test also fails when only the `readBatch` check is removed. Also ran, all passing: the packages `org.apache.spark.sql.execution.datasources.parquet` and `org.apache.spark.sql.execution.datasources.v2.parquet` with `FileBasedDataSourceSuite` (55 suites, 2075 tests passed, 7 ignored), and `SparkThrowableSuite`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (claude-sonnet-5-5)
    
    Closes #59510 from andygrove/SPARK-60262.
    
    Authored-by: Andy Grove <ag...@apache.org>
    Signed-off-by: Peter Toth <pe...@gmail.com>
    (cherry picked from commit f095057ea8de9f33fe816656cf31fbb20897ca3b)
    Signed-off-by: Peter Toth <pe...@gmail.com>
---
 .../src/main/resources/error/error-conditions.json |  6 ++
 docs/sql-migration-guide.md                        |  1 +
 .../spark/sql/errors/QueryExecutionErrors.scala    | 10 +++
 .../parquet/VectorizedColumnReader.java            |  9 +++
 .../datasources/parquet/ParquetReadSupport.scala   | 26 +++++++
 .../parquet/ParquetFieldIdIOSuite.scala            | 89 +++++++++++++++++++++-
 .../parquet/ParquetFieldIdSchemaSuite.scala        | 71 +++++++++++++++++
 .../parquet/ParquetVectorizedSuite.scala           | 34 +++++++++
 8 files changed, 244 insertions(+), 2 deletions(-)

diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index fcedfa4f1a4e..8a2dd3bd44ac 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -6729,6 +6729,12 @@
     },
     "sqlState" : "42846"
   },
+  "PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE" : {
+    "message" : [
+      "Requested fields <requestedFields> resolve to the same Parquet field <parquetField>, which can be read only once. When <fieldIdReadConf> is enabled, a requested field with a field ID is resolved by ID and any other requested field by name. Request the Parquet field only once, and refer to the resulting column more than once in the query instead."
+    ],
+    "sqlState" : "KD000"
+  },
   "PARQUET_TYPE_ILLEGAL" : {
     "message" : [
       "Illegal Parquet type: <parquetType>."
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 6914d962682c..a1865ce57ed7 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -44,6 +44,7 @@ license: |
 - Since Spark 4.4, `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` defaults to `true`. A V2 scan that reports a keyed partitioning but no explicit ordering now also reports itself sorted by its partition key expressions, because every row of such a partition evaluates them to the same value. Partition transforms such as `days(ts)` or `bucket(8, id)` are left out of that ordering. This lets Spark drop a `Sort` it would otherwise place above the scan. An aggregate that groups [...]
 - Since Spark 4.4, `spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled` defaults to `true`. When `GroupPartitionsExec` merges several input partitions that share one partition key value into a single output partition, it now keeps sort orders over the partition key expressions in the ordering it reports, since those expressions are constant within the merged partition. Sort orders over partition transforms such as `days(ts)` or `bucket(8, id)` are left out. Orders over  [...]
 - Since Spark 4.4, the value columns of `UNPIVOT` and `Dataset.unpivot` merge the nested nullability (array element, map value, and struct field nullability) of all unpivoted values. Previously, a value column took the nested nullability of the first listed value, so a `NULL` array element, map value, or struct field of a later value could be returned as a non-`NULL` value such as `0`, and `IS NULL` on it returned `false`. A view created by an earlier Spark version over such an `UNPIVOT` [...]
+- Since Spark 4.4, when `spark.sql.parquet.fieldId.read.enabled` is true, a Parquet read fails with `PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE` if two or more requested fields of a struct (or of the top-level schema) resolve to the same Parquet field. This happens when one requested field is matched by field ID and another by name, or when two requested fields have the same field ID. In Spark 4.3 and earlier, such a read returned wrong values without an error: with the vectorized reader one [...]
 - Since Spark 4.4, when `spark.sql.groupByOrdinal` is true (the default), the pipe `AGGREGATE` operator rejects an aliased integer literal in its `GROUP BY` clause, such as `TABLE t |> AGGREGATE sum(x) AS s GROUP BY 2 AS grp`, with `UNSUPPORTED_FEATURE.PIPE_OPERATOR_AGGREGATE_UNSUPPORTED_CASE`, because a bare integer literal in the same position is an ordinal. Previously the aliased literal was grouped as a constant. Bare ordinals and aliases on other grouping expressions are not affecte [...]
 
 ## Upgrading from Spark SQL 4.2 to 4.3
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index 4c61bf4a7e7a..1600a73e5519 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -1150,6 +1150,16 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
         "matchedFields" -> matchedFields))
   }
 
+  def parquetFieldRequestedMoreThanOnceError(
+      requestedFields: Seq[String], parquetField: String): SparkRuntimeException = {
+    new SparkRuntimeException(
+      errorClass = "PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE",
+      messageParameters = Map(
+        "requestedFields" -> requestedFields.map(f => toSQLId(Seq(f))).mkString(", "),
+        "parquetField" -> toSQLId(Seq(parquetField)),
+        "fieldIdReadConf" -> toSQLConf(SQLConf.PARQUET_FIELD_ID_READ_ENABLED.key)))
+  }
+
   def failedToMergeIncompatibleSchemasError(
       left: StructType, right: StructType, e: Throwable = null): Throwable = {
     new SparkException(
diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
index edb4638eaf22..dfe27c292b7e 100644
--- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
+++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
@@ -237,6 +237,15 @@ public class VectorizedColumnReader {
       if (readState.valuesToReadInPage == 0) {
         int pageValueCount = readPage();
         if (pageValueCount < 0) {
+          if (readState.maxRepetitionLevel == 0) {
+            // A column without repeated values has a value, possibly null, for every row. So
+            // running out of pages with rows left to read means the column chunk holds fewer
+            // values than expected or another reader of the column took its pages, and returning
+            // would leave the rest of the vector unwritten.
+            throw new ParquetDecodingException(String.format(
+              "Column %s ran out of pages with %d of the %d rows of the batch left to read",
+              descriptor, readState.rowsToReadInBatch, total));
+          }
           // we've read all the pages; this could happen when we're reading a repeated list and we
           // don't know where the list will end until we've seen all the pages.
           break;
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala
index e92858f6a1e9..bb41a11ce4ef 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala
@@ -507,6 +507,7 @@ object ParquetReadSupport extends Logging {
         matchCaseInsensitiveField(f)
       }
     }
+    checkNoFieldRequestedMoreThanOnce(structType, clippedType)
     // Ignore MessageType, because it is the root of the schema, not a struct.
     if (returnNullStructIfAllFieldsMissing || !isStructWithMissingAllFields ||
         parquetRecord.isInstanceOf[MessageType]) {
@@ -517,6 +518,31 @@ object ParquetReadSupport extends Logging {
     }
   }
 
+  /**
+   * Fails if two or more requested fields were clipped to the same Parquet field. That happens
+   * when one is looked up by field id and another by name, or when both are looked up by the same
+   * id. The requested Parquet schema would then list the field twice, but a column can be read
+   * only once: the vectorized reader finds no pages left for the second copy and leaves its vector
+   * unwritten, and the parquet-mr reader maps both copies back to the same requested field.
+   *
+   * The fields of a Parquet group have distinct names, so clipped fields with the same name are
+   * the same field.
+   */
+  private def checkNoFieldRequestedMoreThanOnce(
+      structType: StructType,
+      clippedFields: Seq[Type]): Unit = {
+    val clippedNames = new util.HashSet[String]()
+    clippedFields.foreach { clipped =>
+      if (!clippedNames.add(clipped.getName)) {
+        val requestedFields = structType.iterator.zip(clippedFields.iterator)
+          .collect { case (requested, c) if c.getName == clipped.getName => requested.name }
+          .toSeq
+        throw QueryExecutionErrors.parquetFieldRequestedMoreThanOnceError(
+          requestedFields, clipped.getName)
+      }
+    }
+  }
+
   /**
    * Finds the leaf node(s) under a given file schema node that is likely to be cheapest to fetch.
    * Note that multiple leaves can be selected if a map type is deemed to be the cheapest to fetch,
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFieldIdIOSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFieldIdIOSuite.scala
index 7af9ebffb0e7..e4de87a9a3fd 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFieldIdIOSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFieldIdIOSuite.scala
@@ -19,8 +19,8 @@ package org.apache.spark.sql.execution.datasources.parquet
 
 import scala.jdk.CollectionConverters._
 
-import org.apache.spark.SparkException
-import org.apache.spark.sql.Row
+import org.apache.spark.{SparkException, SparkRuntimeException}
+import org.apache.spark.sql.{DataFrame, Row}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSparkSession
 import org.apache.spark.sql.types._
@@ -266,4 +266,89 @@ class ParquetFieldIdIOSuite extends ParquetTest with SharedSparkSession  {
       }
     }
   }
+
+  private def checkFieldRequestedMoreThanOnce(
+      df: => DataFrame,
+      requestedFields: Seq[String],
+      parquetField: String): Unit = {
+    // The reader raises the error on the executor, so it is the cause of the job failure.
+    val cause = intercept[SparkException](df.collect()).getCause
+    checkError(
+      exception = cause.asInstanceOf[SparkRuntimeException],
+      condition = "PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE",
+      parameters = Map(
+        "requestedFields" -> requestedFields.map(f => s"`$f`").mkString(", "),
+        "parquetField" -> s"`$parquetField`",
+        "fieldIdReadConf" -> "\"spark.sql.parquet.fieldId.read.enabled\""))
+  }
+
+  test("SPARK-60262: a field id match and a name match must not select the same column") {
+    withTempDir { dir =>
+      // The file column `A` has field id 1. A requested field with id 1 selects it by id, and a
+      // requested field without an id selects it by name.
+      val writeSchema = new StructType().add("A", IntegerType, true, withId(1))
+      spark.createDataFrame(Seq(Row(1), Row(2), Row(3)).asJava, writeSchema)
+        .write.mode("overwrite").parquet(dir.getCanonicalPath)
+
+      val byId = StructField("x", IntegerType, true, withId(1))
+      Seq("false" -> "a", "true" -> "A").foreach { case (caseSensitive, nameOfA) =>
+        withSQLConf(SQLConf.CASE_SENSITIVE.key -> caseSensitive) {
+          val byName = StructField(nameOfA, IntegerType, true)
+          withAllParquetReaders {
+            Seq(Seq(byId, byName), Seq(byName, byId)).foreach { fields =>
+              checkFieldRequestedMoreThanOnce(
+                spark.read.schema(StructType(fields)).parquet(dir.getCanonicalPath),
+                fields.map(_.name),
+                "A")
+            }
+          }
+        }
+      }
+
+      // A requested field that matches no column does not select the column either.
+      withAllParquetReaders {
+        checkAnswer(
+          spark.read.schema(new StructType().add(byId).add("b", IntegerType, true))
+            .parquet(dir.getCanonicalPath),
+          Seq(Row(1, null), Row(2, null), Row(3, null)))
+      }
+    }
+  }
+
+  test("SPARK-60262: a field id match and a name match must not select the same nested column") {
+    withTempDir { dir =>
+      val writeSchema = new StructType().add(
+        "s", new StructType().add("A", IntegerType, true, withId(2)), true, withId(1))
+      spark.createDataFrame(Seq(Row(Row(1)), Row(Row(2)), Row(Row(3))).asJava, writeSchema)
+        .write.mode("overwrite").parquet(dir.getCanonicalPath)
+
+      val readSchema = new StructType().add(
+        "s",
+        new StructType()
+          .add("x", IntegerType, true, withId(2))
+          .add("a", IntegerType, true),
+        true,
+        withId(1))
+      withAllParquetReaders {
+        checkFieldRequestedMoreThanOnce(
+          spark.read.schema(readSchema).parquet(dir.getCanonicalPath), Seq("x", "a"), "A")
+      }
+    }
+  }
+
+  test("SPARK-60262: two fields with the same field id must not select the same column") {
+    withTempDir { dir =>
+      val writeSchema = new StructType().add("A", IntegerType, true, withId(1))
+      spark.createDataFrame(Seq(Row(1), Row(2), Row(3)).asJava, writeSchema)
+        .write.mode("overwrite").parquet(dir.getCanonicalPath)
+
+      val readSchema = new StructType()
+        .add("x", IntegerType, true, withId(1))
+        .add("y", IntegerType, true, withId(1))
+      withAllParquetReaders {
+        checkFieldRequestedMoreThanOnce(
+          spark.read.schema(readSchema).parquet(dir.getCanonicalPath), Seq("x", "y"), "A")
+      }
+    }
+  }
 }
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFieldIdSchemaSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFieldIdSchemaSuite.scala
index b02820fa1b13..aa49f254ea3a 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFieldIdSchemaSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFieldIdSchemaSuite.scala
@@ -21,6 +21,7 @@ import scala.jdk.CollectionConverters._
 
 import org.apache.parquet.schema.{MessageType, MessageTypeParser}
 
+import org.apache.spark.SparkRuntimeException
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types._
 
@@ -526,4 +527,74 @@ class ParquetFieldIdSchemaSuite extends ParquetSchemaTest {
         |  }
         |}
       """.stripMargin)
+
+  test("SPARK-60262: clipping with field id fails on a Parquet field requested twice") {
+    val fileSchema = MessageTypeParser.parseMessageType(
+      """message root {
+        |  optional int32 A = 1;
+        |  optional group s = 2 {
+        |    optional int32 B = 3;
+        |  }
+        |  optional int32 C = 4;
+        |}
+      """.stripMargin)
+
+    def clip(catalystSchema: StructType, caseSensitive: Boolean = false): MessageType =
+      ParquetReadSupport.clipParquetSchema(
+        fileSchema,
+        catalystSchema,
+        caseSensitive = caseSensitive,
+        useFieldId = true,
+        returnNullStructIfAllFieldsMissing = false)
+
+    def checkFails(
+        catalystSchema: StructType,
+        requestedFields: String,
+        parquetField: String,
+        caseSensitive: Boolean = false): Unit = {
+      checkError(
+        exception = intercept[SparkRuntimeException](clip(catalystSchema, caseSensitive)),
+        condition = "PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE",
+        parameters = Map(
+          "requestedFields" -> requestedFields,
+          "parquetField" -> parquetField,
+          "fieldIdReadConf" -> "\"spark.sql.parquet.fieldId.read.enabled\""))
+    }
+
+    // One field selects the Parquet field by id and the other by name, in either order.
+    checkFails(
+      new StructType().add("x", IntegerType, true, withId(1)).add("a", IntegerType, true),
+      "`x`, `a`", "`A`")
+    checkFails(
+      new StructType().add("a", IntegerType, true).add("x", IntegerType, true, withId(1)),
+      "`a`, `x`", "`A`")
+    checkFails(
+      new StructType().add("x", IntegerType, true, withId(1)).add("A", IntegerType, true),
+      "`x`, `A`", "`A`", caseSensitive = true)
+
+    // Both select it by id, with an unrelated field between them.
+    checkFails(
+      new StructType()
+        .add("x", IntegerType, true, withId(1))
+        .add("c", IntegerType, true)
+        .add("y", IntegerType, true, withId(1)),
+      "`x`, `y`", "`A`")
+    // A third field selects it by name as well, and all three are reported.
+    checkFails(
+      new StructType()
+        .add("x", IntegerType, true, withId(1))
+        .add("y", IntegerType, true, withId(1))
+        .add("a", IntegerType, true),
+      "`x`, `y`, `a`", "`A`")
+
+    // The fields of a nested struct are clipped on their own.
+    checkFails(
+      new StructType().add(
+        "s", new StructType().add("x", IntegerType, true, withId(3)).add("b", IntegerType), true),
+      "`x`, `b`", "`B`")
+
+    // Fields that select different Parquet fields, or none, are fine.
+    clip(new StructType().add("x", IntegerType, true, withId(1)).add("c", IntegerType, true))
+    clip(new StructType().add("x", IntegerType, true, withId(1)).add("z", IntegerType, true))
+  }
 }
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorizedSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorizedSuite.scala
index c1f45b54767a..8611970baa12 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorizedSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorizedSuite.scala
@@ -551,6 +551,40 @@ class ParquetVectorizedSuite extends ParquetTest with SharedSparkSession {
     }
   }
 
+  test("SPARK-60262: a column requested twice fails instead of leaving a vector unwritten") {
+    // The requested schema lists column `a` twice, as it would if two requested fields selected
+    // it, one by field id and one by name. Parquet's own page stores hand out a single page reader
+    // per column, which every reader of that column shares, so the reader of the first copy drains
+    // the pages and the reader of the second finds none. `MemPageStore` would give each its own.
+    val fileSchema = MessageTypeParser.parseMessageType("message root { optional int32 a; }")
+    val requestedSchema = MessageTypeParser.parseMessageType(
+      "message root { optional int32 a; optional int32 a; }")
+    val ty = fileSchema.asGroupType().getType("a").asPrimitiveType()
+    val cd = new ColumnDescriptor(Array("a"), ty, 0, 1)
+    val memPageStore = new MemPageStore(3)
+    // Three non-null values: the definition levels (length 2, an RLE run of 3 x 1), then the
+    // PLAIN int32 values 1, 2 and 3.
+    memPageStore.getPageWriter(cd).writePage(
+      BytesInput.from(Array[Byte](2, 0, 0, 0, 6, 1, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0)), 3, 3,
+      Statistics.createStats(ty), Encoding.RLE, Encoding.RLE, Encoding.PLAIN)
+    val sharedReader = memPageStore.getPageReader(cd)
+    val sharedStore = new PageReadStore {
+      override def getPageReader(descriptor: ColumnDescriptor): PageReader = sharedReader
+      override def getRowCount: Long = 3
+    }
+    val recordReader = new VectorizedParquetRecordReader(
+      DateTimeUtils.getZoneId("EST"), "CORRECTED", "UTC", "CORRECTED", "UTC", true, 3)
+    try {
+      recordReader.initialize(fileSchema, requestedSchema,
+        TestParquetRowGroupReader(Seq(TestPageReadStore(sharedStore, Seq(0L)))), 3)
+      val e = intercept[ParquetDecodingException](recordReader.nextKeyValue())
+      assert(e.getMessage.contains("ran out of pages"), e.getMessage)
+      assert(e.getMessage.contains("[a]"), e.getMessage)
+    } finally {
+      recordReader.close()
+    }
+  }
+
   truncateTypeTest("primitive type", IntegerType, LongType, IntegerType)
 
   truncateTypeTest("basic struct",


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org