You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "rdtr (via GitHub)" <gi...@apache.org> on 2026/10/05 21:45:07 UTC

[PR] [GLUTEN-13180][CORE] Fall back the scan when the ignoreMissingFiles read option differs from the conf [gluten]

rdtr opened a new pull request, #13207:
URL: https://github.com/apache/gluten/pull/13207

   ## What changes are proposed in this pull request?
   
   Related to #13180.
   
   Spark 4.2 has a new test in ParquetQuerySuite: "SPARK-55857: Enabling/disabling ignoreMissingFiles". You can set ignoreMissingFiles in two places:
   
   - the session conf: `spark.sql.files.ignoreMissingFiles`
   - the read option: `spark.read.option("ignoreMissingFiles", ...)`
   
   When both are set, Spark uses the read option (see `FileSourceOptions.ignoreMissingFiles`). Spark works like this since 3.4. So this is an old problem. The new test only made it visible.
   
   ### What happened
   
   I ran the same steps on Gluten:
   
   1. Write two parquet dirs.
   2. Create the DataFrame.
   3. Delete the files in the first dir.
   4. Collect.
   
   | conf | read option | Spark | Gluten (before) |
   |---|---|---|---|
   | true | (none) | skips the missing file | same |
   | false | true | skips the missing file | error: `VeloxRuntimeError ... FILE_NOT_FOUND` |
   | false | (none) | error | same |
   | true | false | error | skips the missing file, no error |
   
   The last row is the worst. The user wants an error. But Gluten gives no error and just returns fewer rows.
   
   ### Why
   
   Gluten sends only the session conf to Velox:
   
   - `GlutenConfig.scala`: `spark.sql.files.ignoreMissingFiles` is sent to native.
   - `cpp/velox/utils/ConfigExtractor.cc`: it is set to `HiveConfig::kIgnoreMissingFilesSession`.
   
   The read option is in `relation.options` (V1) or in `ParquetScan.options` / `OrcScan.options` (V2). Gluten does not look at it. So Velox always uses the session conf.
   
   ### Fix
   
   In Velox, this setting has one value for the whole query. But one query can have two scans with different options. So we cannot just send the option to Velox. For now, this PR falls back the scan:
   
   - `BasicScanExecTransformer`: add `getReadOptions`. It is empty by default. In `doValidateInternal`, get the value in the same way as Spark: `new FileSourceOptions(getReadOptions).ignoreMissingFiles`. If it is different from `SQLConf.get.ignoreMissingFiles`, the validation fails and the scan falls back.
   - `FileSourceScanExecTransformer`: `getReadOptions` returns `relation.options`. Spark gives the same map to `FileScanRDD`.
   - `BatchScanExecTransformer`: `getReadOptions` returns the options of `ParquetScan` / `OrcScan`.
   
   If there is no option, or the option has the same value as the conf, the scan still runs in native. Other scans (Hive, Iceberg, ...) use the empty map, so they do not change.
   
   This fallback is a short-term fix. The real fix should be in Velox. Today `FileSplitReader.cpp` reads ignoreMissingFiles only from the session properties, so each scan cannot have its own value. If `HiveTableHandle` can keep a value for each scan, Gluten can send the read option in the ReadRel. Then we can remove this fallback. I will work on it in a follow-up.
   
   Spark has the same kind of read option for `ignoreCorruptFiles`. This PR does not change it.
   
   ## How was this patch tested?
   
   I added a test "ignoreMissingFiles read option overrides the session conf" to VeloxScanSuite. It checks the four rows in the table above, for both V1 and V2 scans. It also checks that the scan runs in native when there is no option, and falls back when there is an option.
   
   - Spark 4.2 and 4.1 (SPARK_ANSI_SQL_MODE=false): VeloxScanSuite passed (11 tests).
   - I also removed the change in gluten-substrait and ran the test again on Spark 4.2. It failed with `VeloxRuntimeError ... FILE_NOT_FOUND` in the conf=false / option=true case.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-13180][CORE] Fall back the scan when the ignoreMissingFiles read option differs from the conf [gluten]

Posted by "rdtr (via GitHub) via commits" <co...@gluten.apache.org>.
rdtr commented on PR #13207:
URL: https://github.com/apache/gluten/pull/13207#issuecomment-6689656954

   @philo-he could you take a look when you have time? This falls back the scan when the `ignoreMissingFiles` read option differs from `spark.sql.files.ignoreMissingFiles`. Today Gluten ignores the read option. With the conf set to `true`, `spark.read.option("ignoreMissingFiles", "false")` returns fewer rows instead of failing like Spark.
   
   facebookincubator/velox#19840 adds a table parameter that overrides `ignore_missing_files`, and that is the better fix in the long run. Could this go in as an interim fix until it lands? Once it does, I'll replace the fallback with passing the read option to Velox (#13433). I rebased this onto main. The only CI failure is `test that introducing TTL after restart fails query` in `spark-test-spark42 (1)`, the flaky test tracked in #13463.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-13180][CORE] Fall back the scan when the ignoreMissingFiles read option differs from the conf [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13207:
URL: https://github.com/apache/gluten/pull/13207#issuecomment-6003864177

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-13180][CORE] Fall back the scan when the ignoreMissingFiles read option differs from the conf [gluten]

Posted by "rdtr (via GitHub) via commits" <co...@gluten.apache.org>.
rdtr commented on PR #13207:
URL: https://github.com/apache/gluten/pull/13207#issuecomment-6689768697

   @philo-he This is the Gluten side of the ANSI plan agreed in https://github.com/facebookincubator/velox/issues/3869 with mbasmanova and rui-mo (https://github.com/facebookincubator/velox/issues/3869#issuecomment-6641765130). Velox Spark functions whose behavior depends on ANSI read `spark.ansi_enabled` from the session config, and when the mode Spark captured in an expression differs from the session, Gluten falls back to Spark (https://github.com/facebookincubator/velox/issues/3869#issuecomment-6607421212). This PR adds that fallback, and it sets how Gluten handles ANSI from here on, so I'd appreciate your review.
   
   It also fixes a wrong result with the default settings. `spark.gluten.sql.ansiFallback.enabled` looks only at the session, so a view created with ANSI on and read with ANSI off stays native, and `abs` in it wraps around where Spark throws `ARITHMETIC_OVERFLOW`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-13180][CORE] Fall back the scan when the ignoreMissingFiles read option differs from the conf [gluten]

Posted by "philo-he (via GitHub) via commits" <co...@gluten.apache.org>.
philo-he merged PR #13207:
URL: https://github.com/apache/gluten/pull/13207


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-13180][CORE] Fall back the scan when the ignoreMissingFiles read option differs from the conf [gluten]

Posted by "github-actions[bot] (via GitHub) via commits" <co...@gluten.apache.org>.
github-actions[bot] commented on PR #13207:
URL: https://github.com/apache/gluten/pull/13207#issuecomment-6683546505

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-13180][CORE] Fall back the scan when the ignoreMissingFiles read option differs from the conf [gluten]

Posted by "github-actions[bot] (via GitHub) via commits" <co...@gluten.apache.org>.
github-actions[bot] commented on PR #13207:
URL: https://github.com/apache/gluten/pull/13207#issuecomment-6703751486

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org