You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/09/13 05:36:51 UTC

[PR] fix: preserve Spark errors for Parquet timestamp overflow [datafusion-comet]

peterxcli opened a new pull request, #5891:
URL: https://github.com/apache/datafusion-comet/pull/5891

   ## Which issue does this PR close?
   
   Closes #5517.
   
   ## Rationale for this change
   
   Parquet TIMESTAMP_MILLIS overflow currently surfaces as a raw Arrow error. Spark throws a plain `ArithmeticException("long overflow")`, even with ANSI disabled.
   
   ## What changes are included in this PR?
   
   Emit `SparkError::LongOverflow` from checked millisecond-to-microsecond conversion and translate it in the shared JVM error converter. The shared error variant, mapping, and tests match #5457; whichever PR lands second can drop the overlapping additions on rebase.
   
   ## How are these changes tested?
   
   Regression tests check the exact exception class and message for positive and negative overflow, dictionary pages, nested timestamps, and both ANSI settings. Native tests cover JSON serialization and null visibility; existing filtered-scan controls also pass. Focused checks pass on Spark 3.4, 3.5, and 4.1.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve Spark errors for Parquet timestamp overflow [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5891:
URL: https://github.com/apache/datafusion-comet/pull/5891#discussion_r4274439099


##########
spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala:
##########
@@ -69,6 +71,23 @@ class SparkErrorConverterSuite extends AnyFunSuite {
     assert(!ex.getMessage.contains("fallback.parquet"))
   }
 
+  test("ParquetTimestampOverflow wraps the arithmetic cause in a file-read SparkException") {

Review Comment:
   Everything this test asserts is also asserted end to end by `TIMESTAMP_MILLIS overflow fails in native scan` in `ParquetReadSuite`: the error class per Spark version, the `path` parameter, the `ArithmeticException("long overflow")` cause, the two-file task and the `%` and space in the file name. Could this one be dropped together with the `SparkException` and `isSpark40Plus` imports it adds? The counterpoint is that the suite has per-variant converter tests for `CannotReadFile`, so keep it if you prefer that pattern.



##########
spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala:
##########
@@ -339,16 +349,29 @@ abstract class ParquetReadSuite extends CometTestBase {
           }
 
           Seq(false, true).foreach { ansiEnabled =>
-            withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
-              readParquetFile(path.toString) { df =>
-                Seq("ts", "ts_ntz", "s", "s.ts", "s.ts_ntz", "a", "m").foreach { column =>
-                  val selected = df.select(column)
-                  assert(collect(selected.queryExecution.executedPlan) {
-                    case _: CometNativeScanExec => true
-                  }.nonEmpty)
+            withSQLConf(
+              SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString,
+              SQLConf.FILES_MIN_PARTITION_NUM.key -> "1",
+              SQLConf.FILES_MAX_PARTITION_BYTES.key -> "134217728") {

Review Comment:
   This suite already has `forceSinglePartitionConf` (line 960), which sets `FILES_MIN_PARTITION_NUM` to 1 and documents why. Could the test use it? `134217728` is Spark's default for `spark.sql.files.maxPartitionBytes` (`128MB`). I could not find anything in the test base classes that overrides it, so that line can go.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve Spark errors for Parquet timestamp overflow [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5891:
URL: https://github.com/apache/datafusion-comet/pull/5891#issuecomment-5826012972

   @peterxcli could you fix merge conflict of this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve Spark errors for Parquet timestamp overflow [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5891:
URL: https://github.com/apache/datafusion-comet/pull/5891#issuecomment-5955531433

   I would like to do a deeper review of this PR before it is merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve Spark errors for Parquet timestamp overflow [datafusion-comet]

Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #5891:
URL: https://github.com/apache/datafusion-comet/pull/5891#discussion_r4017108549


##########
native/core/src/parquet/parquet_exec.rs:
##########
@@ -221,6 +222,11 @@ pub(crate) fn init_datasource_exec(
         _ => Arc::new(parquet_source),
     };
 
+    let file_source = if spark_parquet_options.checked_timestamp_overflow {
+        ParquetErrorContext::wrap(file_source)

Review Comment:
   This wrapper makes `downcast_to_file_source::<ParquetSource>()` in `dynamic_filter.rs` return `None`, so wrapped scans lose runtime reader-filter pushdown. That causes the [Rust CI failure](https://github.com/apache/datafusion-comet/actions/runs/34953412077/job/104331959346).
   
   I reproduced it locally; recognizing `ParquetErrorContext` in that type check makes the existing test pass. Please update the check while preserving this wrapper.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve Spark errors for Parquet timestamp overflow [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #5891:
URL: https://github.com/apache/datafusion-comet/pull/5891#issuecomment-5677969058

   @sunchao addressed review! ptal again, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve Spark errors for Parquet timestamp overflow [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5891:
URL: https://github.com/apache/datafusion-comet/pull/5891#discussion_r4010865700


##########
spark/src/main/scala/org/apache/comet/SparkErrorConverter.scala:
##########
@@ -86,11 +86,10 @@ object SparkErrorConverter extends ShimSparkErrorConverter {
     val json = parse(e.getMessage)
     val errorJson = json.extract[ErrorJson]
     val rawParams = errorJson.params.getOrElse(Map.empty)
-    // CannotReadFile carries the offending file path natively only for the object_store NotFound
-    // case; for corrupt/truncated parquet the native error has no path, so fall back to the
-    // per-task file list threaded in from CometExecIterator.
+    // File-read errors without a native path use the per-task file list from CometExecIterator.
     val params =
-      if (errorJson.errorType == "CannotReadFile"
+      if ((errorJson.errorType == "CannotReadFile" ||
+          errorJson.errorType == "ParquetTimestampOverflow")

Review Comment:
   ### Correctness
   
   [P2] Preserve the failing file path for multi-file tasks
   
   Could we carry the active Parquet file's path with `ParquetTimestampOverflow` instead of filling it from the whole task list? `CometNativeScanExec` populates that list from every file in the `FilePartition`, and normal bin packing can put several files in one task. With one healthy file and one overflowing file, this branch sets Spark's `path` parameter to `file:///data/good.parquet,file:///data/bad.parquet`, rather than the single failing file that Spark's `FileScanRDD` supplies. I reproduced this with the exact current converter and 4.x shim against Spark 4.0.4. The new native error has no path parameter, so it always needs this fallback. The existing tests use one file and cannot catch this case. Please preserve the failing path at the native reader boundary and add a two-file, single-task regression, including the fused shuffle path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve Spark errors for Parquet timestamp overflow [datafusion-comet]

Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #5891:
URL: https://github.com/apache/datafusion-comet/pull/5891#discussion_r4017108549


##########
native/core/src/parquet/parquet_exec.rs:
##########
@@ -221,6 +222,11 @@ pub(crate) fn init_datasource_exec(
         _ => Arc::new(parquet_source),
     };
 
+    let file_source = if spark_parquet_options.checked_timestamp_overflow {
+        ParquetErrorContext::wrap(file_source)

Review Comment:
   Please update the `ParquetSource` check in `try_attach_parquet_reader_filter` to recognize this wrapper. Otherwise wrapped scans skip reader-filter pushdown, causing the [Rust CI failure](https://github.com/apache/datafusion-comet/actions/runs/34953412077/job/104331959346).
   
   I reproduced the failure locally; recognizing `ParquetErrorContext` in that check makes the same test pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve Spark errors for Parquet timestamp overflow [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5891:
URL: https://github.com/apache/datafusion-comet/pull/5891#issuecomment-6361377901

   Thanks @peterxcli, this is still approved. The branch conflicts with main now, so could you rebase? Once CI is green it can go into the merge queue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve Spark errors for Parquet timestamp overflow [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5891:
URL: https://github.com/apache/datafusion-comet/pull/5891


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org