You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/07 16:47:20 UTC
(datafusion-comet) 01/01: fix: keep a Spark scan unconverted when the plan reads input_file_name (#6703)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6703-9b54a0c7d8cb7724edee9eba303561de0a51481d
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 10d958fa08dacba12288bc3c442dbab7b4839744
Author: KAZUYUKI TANIMURA <kt...@apple.com>
AuthorDate: Wed Oct 7 16:46:56 2026 +0000
fix: keep a Spark scan unconverted when the plan reads input_file_name (#6703)
* fix: keep a Spark scan unconverted when the plan reads input_file_name
input_file_name, input_file_block_start and input_file_block_length read
InputFileBlockHolder, a thread-local that the reader producing the rows
sets as it moves from file to file. CometScanRule already keeps Spark's
scan when a plan uses them, but with spark.comet.convert.parquet.enabled
CometExecRule then wrapped that scan in CometSparkToColumnarExec. The
conversion fills a batch and Comet operators above it pull more before
they emit, so by the time the Spark project evaluated the expressions
the reader had moved to a later file or unset the holder. Rows reported
another file's values or "", -1, -1.
Move CometScanRule's check into a shared helper and refuse to convert any
leaf to Arrow when the plan reads these expressions, recording a
fallback reason. This covers the Parquet, CSV and JSON conversions for
V1 and V2 scans, and other sparkToColumnar leaves such as RDDScan, whose
HadoopRDD sets the same holder.
Closes #6573.
* fix: keep a typed Dataset's output unconverted when the plan reads input_file_name
The typed Dataset conversion from upstream converts the rows a
SerializeFromObjectExec produces to Arrow. Like the conversion of a
leaf, it fills a batch before the operators above it see a row, so it
reads ahead of the file that a Spark project evaluating input_file_name
expects. Apply the same fallback to it, sharing the reason with the leaf
conversion.
Also switch the RDD test to spark.comet.convert.rdd.enabled, which
replaced the deprecated spark.comet.sparkToColumnar setting for RDDScan,
and mention typed Dataset output in the user guide note.
* docs: note the input_file_name fallback for typed Dataset output in the operator guide
---
.../contributor-guide/adding_a_new_operator.md | 5 ++-
docs/source/user-guide/latest/datasources.md | 5 +++
.../org/apache/comet/rules/CometExecRule.scala | 40 +++++++++++++++++--
.../org/apache/comet/rules/CometScanRule.scala | 26 ++++++++----
.../org/apache/comet/exec/CometExecSuite.scala | 46 ++++++++++++++++++++++
.../apache/comet/exec/CometTypedDatasetSuite.scala | 20 ++++++++++
6 files changed, 129 insertions(+), 13 deletions(-)
diff --git a/docs/source/contributor-guide/adding_a_new_operator.md b/docs/source/contributor-guide/adding_a_new_operator.md
index 09cac3a1c7..85946f44fd 100644
--- a/docs/source/contributor-guide/adding_a_new_operator.md
+++ b/docs/source/contributor-guide/adding_a_new_operator.md
@@ -148,7 +148,10 @@ own operators itself. Spark computes a typed operation's rows one at a time, as
the conversion fills a whole Arrow batch first. So the rule leaves the output unconverted where a
limit, a `mapPartitions` function, or code reading `Dataset.rdd` could stop reading it early, unless
an operator that reads all of its input first, such as an exchange, a sort, or a hash aggregate,
-sits in between. Fusing the deserializer, the `Invoke` that calls the user function, and the
+sits in between. For the same reason, it leaves the output unconverted when the plan uses
+`input_file_name()`, `input_file_block_start()` or `input_file_block_length()`: filling the batch
+moves the scan's reader past the file that these report when Spark evaluates them above the
+conversion. Fusing the deserializer, the `Invoke` that calls the user function, and the
serializer of `Dataset.map` into one projection in the JVM codegen dispatcher was tried in
[#5714](https://github.com/apache/datafusion-comet/pull/5714) and dropped. The dispatcher only calls
into Spark's own classes, and the conversion gets nearly the same speedup for `map` while also
diff --git a/docs/source/user-guide/latest/datasources.md b/docs/source/user-guide/latest/datasources.md
index eb56d1e897..8682633d71 100644
--- a/docs/source/user-guide/latest/datasources.md
+++ b/docs/source/user-guide/latest/datasources.md
@@ -87,6 +87,11 @@ The same types apply to the output of typed `Dataset` operations, such as `map`,
converts when `spark.comet.convert.typedDataset.enabled=true`. A column of any other type keeps
the operators above the typed operation on Spark.
+Comet does not convert a source, or the output of a typed `Dataset` operation, when the query uses
+`input_file_name()`, `input_file_block_start()` or `input_file_block_length()`, so the operators
+above it run in Spark. These functions report the file that Spark's reader is on when Spark evaluates
+them, and the conversion and the Comet operators above it would read ahead of that.
+
## Data Catalogs
### Apache Iceberg
diff --git a/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala b/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala
index 4327256dbb..fdf37b4e2b 100644
--- a/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala
+++ b/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala
@@ -186,6 +186,11 @@ object CometExecRule {
*/
private val TYPED_DATASET_PARTIAL_READER: TreeNodeTag[String] =
TreeNodeTag[String]("comet.typedDatasetPartialReader")
+
+ /** Why Comet does not convert rows to Arrow in a plan that reads `InputFileBlockHolder`. */
+ private val INPUT_FILE_BLOCK_FALLBACK_REASON: String =
+ "Spark to Arrow conversion is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length"
}
/**
@@ -409,6 +414,9 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
*/
// spotless:on
private def transform(plan: SparkPlan): SparkPlan = {
+ // Walks the whole plan, so it is lazy: only consulted once a node could be converted.
+ lazy val readsInputFileBlock = CometScanRule.readsInputFileBlock(plan)
+
if (CometConf.COMET_CONVERT_FROM_TYPED_DATASET_ENABLED.get(conf)) {
tagPartiallyReadTypedDatasetOutputs(plan)
}
@@ -484,7 +492,7 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
s"Set ${CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED.key}=true to enable it.")
}
- if (shouldApplySparkToColumnar(conf, scan)) {
+ if (shouldApplySparkToColumnar(conf, scan, readsInputFileBlock)) {
convertToComet(scan, CometSparkToColumnarExec).getOrElse(scan)
} else {
scan
@@ -506,7 +514,7 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
case op: LeafExecNode if hasEnabledHandler(op) =>
convertToComet(op, allExecs(op.getClass))
.orElse {
- if (shouldApplySparkToColumnar(conf, op)) {
+ if (shouldApplySparkToColumnar(conf, op, readsInputFileBlock)) {
convertToComet(op, CometSparkToColumnarExec)
} else {
None
@@ -514,7 +522,7 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
}
.getOrElse(op)
- case op if shouldApplySparkToColumnar(conf, op) =>
+ case op if shouldApplySparkToColumnar(conf, op, readsInputFileBlock) =>
convertToComet(op, CometSparkToColumnarExec).getOrElse(op)
// Typed Dataset operations (`map`, `flatMap`, `mapPartitions`, `mapGroups`, ...) pass JVM
@@ -530,6 +538,10 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
"Comet does not convert the output of a typed Dataset operation when " +
s"$reader can stop reading it early, because filling an Arrow batch would " +
"run the user function on rows that Spark never reaches")
+ // The conversion reads ahead of input_file_name and friends, as it does over a leaf.
+ // See shouldApplySparkToColumnar.
+ case None if readsInputFileBlock =>
+ withFallbackReason(op, CometExecRule.INPUT_FILE_BLOCK_FALLBACK_REASON)
case None =>
convertTypedDatasetOutput(op)
}
@@ -1420,7 +1432,27 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
convertToComet(s, CometShuffleExchangeExec).getOrElse(preserveSparkAggregateBuffers(s)))
}
- private def shouldApplySparkToColumnar(conf: SQLConf, op: SparkPlan): Boolean = {
+ private def shouldApplySparkToColumnar(
+ conf: SQLConf,
+ op: SparkPlan,
+ readsInputFileBlock: => Boolean): Boolean = {
+ // A converted leaf reads ahead of the Spark operator that evaluates input_file_name and
+ // friends: the conversion fills a whole batch, and Comet operators above it may pull more
+ // before they emit. By the time Spark evaluates them, the leaf's reader may have moved on to
+ // a later file or unset InputFileBlockHolder at the end of its input, so rows would report
+ // another file's values or the unset defaults. Leave the leaf on Spark so that the plan above
+ // it stays on Spark too.
+ if (!canApplySparkToColumnar(conf, op)) {
+ false
+ } else if (readsInputFileBlock) {
+ withFallbackReason(op, CometExecRule.INPUT_FILE_BLOCK_FALLBACK_REASON)
+ false
+ } else {
+ true
+ }
+ }
+
+ private def canApplySparkToColumnar(conf: SQLConf, op: SparkPlan): Boolean = {
// Only consider converting leaf nodes to columnar currently, so that all the following
// operators can have a chance to be converted to columnar. Leaf operators that output
// columnar batches, such as Spark's vectorized readers, will also be converted to native
diff --git a/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala b/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala
index b109fa158e..b80136314b 100644
--- a/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala
+++ b/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala
@@ -354,14 +354,10 @@ case class CometScanRule(session: SparkSession)
withFallbackReason(scanExec, "Native Parquet Variant scans do not support encryption")
return None
}
- // input_file_name, input_file_block_start, and input_file_block_length read from
- // InputFileBlockHolder, a thread-local set by Spark's FileScanRDD. The native DataFusion
- // scan does not use FileScanRDD, so these expressions would return empty/default values.
- if (plan.exists(node =>
- node.expressions.exists(_.exists {
- case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
- case _ => false
- }))) {
+ // input_file_name, input_file_block_start, and input_file_block_length read values that
+ // Spark's FileScanRDD sets. The native DataFusion scan does not use FileScanRDD, so these
+ // expressions would return empty/default values.
+ if (CometScanRule.readsInputFileBlock(plan)) {
withFallbackReason(
scanExec,
"Native Parquet scan is not compatible with input_file_name, " +
@@ -1185,6 +1181,20 @@ case class CometScanTypeChecker() extends DataTypeSupport with CometTypeShim {
object CometScanRule extends Logging {
+ /**
+ * Whether any node in `plan` evaluates `input_file_name`, `input_file_block_start` or
+ * `input_file_block_length`. These read `InputFileBlockHolder`, a thread-local that the reader
+ * producing the rows (`FileScanRDD`, the V2 file readers, `HadoopRDD`, `NewHadoopRDD`,
+ * connector readers such as Iceberg's) sets as it moves from file to file, so they only return
+ * each row's values when Spark evaluates them as that reader produces the row.
+ */
+ def readsInputFileBlock(plan: SparkPlan): Boolean =
+ plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))
+
// Memo of `NativeBase.isObjectStoreSchemeSupported`, keyed by the probe URL rather than the
// scheme: object_store's parser keys on (scheme, host-presence), so an authorityless URL would
// otherwise poison the authority-bearing form of the same scheme.
diff --git a/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala
index 465da3cdd9..478c62f4ed 100644
--- a/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala
@@ -4713,6 +4713,52 @@ class CometExecSuite extends CometTestBase {
}
}
+ // https://github.com/apache/datafusion-comet/issues/6573
+ private val inputFileColumns =
+ Seq("input_file_name()", "input_file_block_start()", "input_file_block_length()", "id")
+ private val inputFileFallbackReason =
+ "Spark to Arrow conversion is not compatible with input_file_name"
+
+ test("input_file_name above a converted Spark file scan keeps the scan on Spark") {
+ for (format <- Seq("parquet", "csv", "json"); v1List <- Seq("", format)) {
+ withTempPath { dir =>
+ spark.range(9000).repartition(3).write.format(format).save(dir.toString)
+ withSQLConf(
+ SQLConf.USE_V1_SOURCE_LIST.key -> v1List,
+ CometConf.COMET_CONVERT_FROM_PARQUET_ENABLED.key -> "true",
+ CometConf.COMET_CONVERT_FROM_CSV_ENABLED.key -> "true",
+ CometConf.COMET_CONVERT_FROM_JSON_ENABLED.key -> "true") {
+ // The filter would run in Comet above the conversion. Without a Comet operator there,
+ // Comet drops the conversion and Spark's project reads the scan directly.
+ val df = spark.read
+ .schema("id LONG")
+ .format(format)
+ .load(dir.toString)
+ .where("id >= 0")
+ .selectExpr(inputFileColumns: _*)
+ val (_, cometPlan) = checkSparkAnswerAndFallbackReason(df, inputFileFallbackReason)
+ assert(collect(cometPlan) { case c: CometSparkToColumnarExec => c }.isEmpty)
+ }
+ }
+ }
+ }
+
+ test("input_file_name above a converted RDD scan keeps the scan on Spark") {
+ withTempPath { dir =>
+ spark.range(9000).repartition(3).selectExpr("CAST(id AS STRING)").write.text(dir.toString)
+ withSQLConf(CometConf.COMET_CONVERT_FROM_RDD_ENABLED.key -> "true") {
+ // HadoopRDD sets the file Spark evaluates input_file_name against, as FileScanRDD does.
+ val rows = spark.sparkContext.textFile(dir.toString).map(line => Row(line.toLong))
+ val df = spark
+ .createDataFrame(rows, new StructType().add("id", LongType))
+ .where("id >= 0")
+ .selectExpr(inputFileColumns: _*)
+ val (_, cometPlan) = checkSparkAnswerAndFallbackReason(df, inputFileFallbackReason)
+ assert(collect(cometPlan) { case c: CometSparkToColumnarExec => c }.isEmpty)
+ }
+ }
+ }
+
test("read CSV file") {
Seq("", "csv").foreach { v1List =>
withSQLConf(
diff --git a/spark/src/test/scala/org/apache/comet/exec/CometTypedDatasetSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometTypedDatasetSuite.scala
index 9b809e24e8..c9e1a349e3 100644
--- a/spark/src/test/scala/org/apache/comet/exec/CometTypedDatasetSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/exec/CometTypedDatasetSuite.scala
@@ -340,6 +340,26 @@ class CometTypedDatasetSuite extends CometTestBase {
}
}
+ // https://github.com/apache/datafusion-comet/issues/6573
+ convertTest("input_file_name above a map keeps the map's output on Spark") {
+ withTempPath { dir =>
+ spark.range(9000).repartition(3).write.parquet(dir.toString)
+ // The filter would run in Comet above the conversion, which would read ahead of the files
+ // Spark's reader is on when Spark evaluates input_file_name in the project.
+ val df = spark.read
+ .parquet(dir.toString)
+ .as[Long]
+ .map(_ + 1)
+ .toDF("id")
+ .where("id >= 0")
+ .selectExpr("input_file_name()", "input_file_block_start()", "input_file_block_length()")
+ val (_, plan) = checkSparkAnswerAndFallbackReason(
+ df,
+ "Spark to Arrow conversion is not compatible with input_file_name")
+ assert(conversions(plan).isEmpty, plan)
+ }
+ }
+
test("off by default") {
withRecs() { ds =>
val (_, plan) = checkSparkAnswer(ds.map(r => TypedDsRec(r.a + 1, r.b)).groupBy("b").count())
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org