You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "parthchandra (via GitHub)" <gi...@apache.org> on 2025/06/18 23:13:27 UTC

[PR] chore: add a test case to read from an arbitrarily complex type schema [datafusion-comet]

parthchandra opened a new pull request, #1911:
URL: https://github.com/apache/datafusion-comet/pull/1911

   ## Which issue does this PR close?
   
   Adds a new unit test. Also adds a method to generate a complex type parquet file that can be used to test various complex type cases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add a test case to read from an arbitrarily complex type schema [datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1911:
URL: https://github.com/apache/datafusion-comet/pull/1911#issuecomment-2986043661

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1911?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 32.76%. Comparing base [(`f09f8af`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`9d480a4`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/9d480a4bcf8dfe5d0ad709c947c267898b655bc2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 269 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##               main    #1911       +/-   ##
   =============================================
   - Coverage     56.12%   32.76%   -23.37%     
   + Complexity      976      798      -178     
   =============================================
     Files           119      130       +11     
     Lines         11743    12746     +1003     
     Branches       2251     2405      +154     
   =============================================
   - Hits           6591     4176     -2415     
   - Misses         4012     7599     +3587     
   + Partials       1140      971      -169     
   ```
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/1911?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add a test case to read from an arbitrarily complex type schema [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #1911:
URL: https://github.com/apache/datafusion-comet/pull/1911#discussion_r2155755872


##########
spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala:
##########
@@ -1946,6 +1946,52 @@ class ParquetReadV1Suite extends ParquetReadSuite with AdaptiveSparkPlanHelper {
       }
     }
   }
+
+  test("read basic complex types") {
+    Seq(true, false).foreach(dictionaryEnabled => {
+      withTempPath { dir =>
+        val path = new Path(dir.toURI.toString, "complex_types.parquet")
+        makeParquetFileComplexTypes(path, dictionaryEnabled, 10)
+        withParquetTable(path.toUri.toString, "complex_types") {
+          Seq(CometConf.SCAN_NATIVE_DATAFUSION, CometConf.SCAN_NATIVE_ICEBERG_COMPAT).foreach(
+            scanMode => {
+              withSQLConf(CometConf.COMET_NATIVE_SCAN_IMPL.key -> scanMode) {
+                checkSparkAnswer(sql("select * from complex_types"))

Review Comment:
   Perhaps we can create a ticket to include `map_entries` in this test once it is supported?



##########
spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala:
##########
@@ -1946,6 +1946,52 @@ class ParquetReadV1Suite extends ParquetReadSuite with AdaptiveSparkPlanHelper {
       }
     }
   }
+
+  test("read basic complex types") {
+    Seq(true, false).foreach(dictionaryEnabled => {
+      withTempPath { dir =>
+        val path = new Path(dir.toURI.toString, "complex_types.parquet")
+        makeParquetFileComplexTypes(path, dictionaryEnabled, 10)
+        withParquetTable(path.toUri.toString, "complex_types") {
+          Seq(CometConf.SCAN_NATIVE_DATAFUSION, CometConf.SCAN_NATIVE_ICEBERG_COMPAT).foreach(
+            scanMode => {
+              withSQLConf(CometConf.COMET_NATIVE_SCAN_IMPL.key -> scanMode) {
+                checkSparkAnswer(sql("select * from complex_types"))

Review Comment:
   Thanks. Perhaps we can create a ticket to include `map_entries` in this test once it is supported?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add a test case to read from an arbitrarily complex type schema [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #1911:
URL: https://github.com/apache/datafusion-comet/pull/1911#issuecomment-2989423116

   Merged. Thank you for the review @kazuyukitanimura 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add a test case to read from an arbitrarily complex type schema [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #1911:
URL: https://github.com/apache/datafusion-comet/pull/1911#discussion_r2157793226


##########
spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala:
##########
@@ -1946,6 +1946,52 @@ class ParquetReadV1Suite extends ParquetReadSuite with AdaptiveSparkPlanHelper {
       }
     }
   }
+
+  test("read basic complex types") {
+    Seq(true, false).foreach(dictionaryEnabled => {
+      withTempPath { dir =>
+        val path = new Path(dir.toURI.toString, "complex_types.parquet")
+        makeParquetFileComplexTypes(path, dictionaryEnabled, 10)
+        withParquetTable(path.toUri.toString, "complex_types") {
+          Seq(CometConf.SCAN_NATIVE_DATAFUSION, CometConf.SCAN_NATIVE_ICEBERG_COMPAT).foreach(
+            scanMode => {
+              withSQLConf(CometConf.COMET_NATIVE_SCAN_IMPL.key -> scanMode) {
+                checkSparkAnswer(sql("select * from complex_types"))

Review Comment:
   I don't think map_entries is important for the test since map_keys and map_values manage to extract the same fields as well. But added an issue anyway - 
   https://github.com/apache/datafusion-comet/issues/1916



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add a test case to read from an arbitrarily complex type schema [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #1911:
URL: https://github.com/apache/datafusion-comet/pull/1911#discussion_r2155719528


##########
spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala:
##########
@@ -1946,6 +1946,52 @@ class ParquetReadV1Suite extends ParquetReadSuite with AdaptiveSparkPlanHelper {
       }
     }
   }
+
+  test("read basic complex types") {
+    Seq(true, false).foreach(dictionaryEnabled => {
+      withTempPath { dir =>
+        val path = new Path(dir.toURI.toString, "complex_types.parquet")
+        makeParquetFileComplexTypes(path, dictionaryEnabled, 10)
+        withParquetTable(path.toUri.toString, "complex_types") {
+          Seq(CometConf.SCAN_NATIVE_DATAFUSION, CometConf.SCAN_NATIVE_ICEBERG_COMPAT).foreach(
+            scanMode => {
+              withSQLConf(CometConf.COMET_NATIVE_SCAN_IMPL.key -> scanMode) {
+                checkSparkAnswer(sql("select * from complex_types"))

Review Comment:
   What about using checkSparkAnswerAndOperator in order to make sure it is not falling back?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add a test case to read from an arbitrarily complex type schema [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #1911:
URL: https://github.com/apache/datafusion-comet/pull/1911#discussion_r2155736023


##########
spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala:
##########
@@ -1946,6 +1946,52 @@ class ParquetReadV1Suite extends ParquetReadSuite with AdaptiveSparkPlanHelper {
       }
     }
   }
+
+  test("read basic complex types") {
+    Seq(true, false).foreach(dictionaryEnabled => {
+      withTempPath { dir =>
+        val path = new Path(dir.toURI.toString, "complex_types.parquet")
+        makeParquetFileComplexTypes(path, dictionaryEnabled, 10)
+        withParquetTable(path.toUri.toString, "complex_types") {
+          Seq(CometConf.SCAN_NATIVE_DATAFUSION, CometConf.SCAN_NATIVE_ICEBERG_COMPAT).foreach(
+            scanMode => {
+              withSQLConf(CometConf.COMET_NATIVE_SCAN_IMPL.key -> scanMode) {
+                checkSparkAnswer(sql("select * from complex_types"))

Review Comment:
   Done. Had to modify the query to not use `map_entries` (which is not implemented at this time)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add a test case to read from an arbitrarily complex type schema [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra merged PR #1911:
URL: https://github.com/apache/datafusion-comet/pull/1911


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org