You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "weimingdiit (via GitHub)" <gi...@apache.org> on 2026/05/16 04:01:15 UTC

[PR] [AURON #2273] Fallback Hudi incremental queries from native scan conversion [auron]

weimingdiit opened a new pull request, #2274:
URL: https://github.com/apache/auron/pull/2274

   
   # Which issue does this PR close?
   
   Closes #2273 
   
   # Rationale for this change
   
   Hudi incremental queries depend on Hudi timeline and instant filtering semantics. Native Hudi scan currently converts supported Hudi file scans to native Parquet/ORC scans, but native scan does not implement incremental query semantics. These queries should fallback to Spark/Hudi until incremental scan support is explicitly added.
   
   # What changes are included in this PR?
   
   - Detect incremental Hudi query options in `HudiScanSupport`.
   - Fallback native scan conversion when:
     - `hoodie.datasource.query.type=incremental`
     - `hoodie.datasource.read.begin.instanttime`
     - `hoodie.datasource.read.end.instanttime`
   - Add unit coverage for incremental query options, including case-insensitive option keys.
   
   # Are there any user-facing changes?
   Yes. Hudi incremental queries will stay on Spark/Hudi scan instead of being converted to native scan.
   
   # How was this patch tested?
   - Added unit coverage in `HudiScanSupportSuite`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [AURON #2273] Fallback Hudi incremental queries from native scan conversion [auron]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 merged PR #2274:
URL: https://github.com/apache/auron/pull/2274


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [AURON #2273] Fallback Hudi incremental queries from native scan conversion [auron]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #2274:
URL: https://github.com/apache/auron/pull/2274#issuecomment-4524082984

   @weimingdiit Thanks for the contribution! @merrily01 @yew1eb Thanks for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [AURON #2273] Fallback Hudi incremental queries from native scan conversion [auron]

Posted by "yew1eb (via GitHub)" <gi...@apache.org>.
yew1eb commented on code in PR #2274:
URL: https://github.com/apache/auron/pull/2274#discussion_r3252428152


##########
thirdparty/auron-hudi/src/test/scala/org/apache/spark/sql/auron/hudi/HudiScanSupportSuite.scala:
##########
@@ -284,6 +284,29 @@ class HudiScanSupportSuite extends SparkFunSuite with SharedSparkSession {
           "hoodie.datasource.query.type" -> "snapshot")))
   }
 
+  test("hudi isSupported rejects incremental query options") {
+    val fileFormatName =
+      "org.apache.spark.sql.execution.datasources.parquet.HoodieParquetFileFormat"
+    val cowOptions = Map("hoodie.datasource.write.table.type" -> "COPY_ON_WRITE")
+
+    assert(
+      !HudiScanSupport.isSupported(
+        fileFormatName,
+        cowOptions + ("hoodie.datasource.query.type" -> "incremental")))
+    assert(
+      !HudiScanSupport.isSupported(
+        fileFormatName,
+        cowOptions + ("hoodie.datasource.read.begin.instanttime" -> "20240101010101")))
+    assert(
+      !HudiScanSupport.isSupported(
+        fileFormatName,
+        cowOptions + ("hoodie.datasource.read.end.instanttime" -> "20240102010101")))
+    assert(
+      !HudiScanSupport.isSupported(
+        fileFormatName,
+        cowOptions + ("Hoodie.DataSource.Read.Begin.InstantTime" -> "20240101010101")))

Review Comment:
   Consider adding a case for the legacy `hoodie.datasource.view.type=incremental` key — `queryTypeFromOptions` checks both keys, so it already works, but an explicit assertion would document the behavior:
    `assert(!HudiScanSupport.isSupported(fileFormatName, cowOptions + ("hoodie.datasource.view.type" -> "incremental")))`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [AURON #2273] Fallback Hudi incremental queries from native scan conversion [auron]

Posted by "weimingdiit (via GitHub)" <gi...@apache.org>.
weimingdiit commented on code in PR #2274:
URL: https://github.com/apache/auron/pull/2274#discussion_r3258897743


##########
thirdparty/auron-hudi/src/test/scala/org/apache/spark/sql/auron/hudi/HudiScanSupportSuite.scala:
##########
@@ -284,6 +284,29 @@ class HudiScanSupportSuite extends SparkFunSuite with SharedSparkSession {
           "hoodie.datasource.query.type" -> "snapshot")))
   }
 
+  test("hudi isSupported rejects incremental query options") {
+    val fileFormatName =
+      "org.apache.spark.sql.execution.datasources.parquet.HoodieParquetFileFormat"
+    val cowOptions = Map("hoodie.datasource.write.table.type" -> "COPY_ON_WRITE")
+
+    assert(
+      !HudiScanSupport.isSupported(
+        fileFormatName,
+        cowOptions + ("hoodie.datasource.query.type" -> "incremental")))
+    assert(
+      !HudiScanSupport.isSupported(
+        fileFormatName,
+        cowOptions + ("hoodie.datasource.read.begin.instanttime" -> "20240101010101")))
+    assert(
+      !HudiScanSupport.isSupported(
+        fileFormatName,
+        cowOptions + ("hoodie.datasource.read.end.instanttime" -> "20240102010101")))
+    assert(
+      !HudiScanSupport.isSupported(
+        fileFormatName,
+        cowOptions + ("Hoodie.DataSource.Read.Begin.InstantTime" -> "20240101010101")))

Review Comment:
   Thanks for the suggestion. I added an explicit assertion for the legacy `hoodie.datasource.view.type=incremental` option to document that it also falls back from native scan conversion.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org