You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kazantsev-maksim (via GitHub)" <gi...@apache.org> on 2025/06/25 18:45:19 UTC

[PR] Feat: support map_from_arrays [datafusion-comet]

kazantsev-maksim opened a new pull request, #1932:
URL: https://github.com/apache/datafusion-comet/pull/1932

   ## Which issue does this PR close?
   
   Related to Epic: https://github.com/apache/datafusion-comet/issues/1044
   Spark: https://spark.apache.org/docs/latest/api/sql/index.html#map_from_arrays
   
   ## Rationale for this change
   
   Defined under Epic: https://github.com/apache/datafusion-comet/issues/1044
   
   ## What changes are included in this PR?
   
   1. maps.scala - added all already implement map funcs
   2. QueryPlanSerde.scala - map_from_arrays pattern matching case has been added and micro refactoring
   
   ## How are these changes tested?
   
   A new UT has been added.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Feat: support map_from_arrays [datafusion-comet]

Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on code in PR #1932:
URL: https://github.com/apache/datafusion-comet/pull/1932#discussion_r2173441584


##########
spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import scala.util.Random
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.internal.SQLConf
+
+import org.apache.comet.testing.{DataGenOptions, ParquetGenerator}
+
+class CometMapExpressionSuite extends CometTestBase {

Review Comment:
   fixed!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Feat: support map_from_arrays [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1932:
URL: https://github.com/apache/datafusion-comet/pull/1932#discussion_r2173421986


##########
spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import scala.util.Random
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.internal.SQLConf
+
+import org.apache.comet.testing.{DataGenOptions, ParquetGenerator}
+
+class CometMapExpressionSuite extends CometTestBase {

Review Comment:
   This new suite will need to be added in `.github/workflows/pr_build_linux.yml` and `.github/workflows/pr_build_macos.yml` before it can run in CI. I noticed that the `CometBitwiseExpressionSuite` suite also needs to be added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Feat: support map_from_arrays [datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1932:
URL: https://github.com/apache/datafusion-comet/pull/1932#issuecomment-3005990602

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1932?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `73.33333%` with `8 lines` in your changes missing coverage. Please review.
   > Project coverage is 33.60%. Comparing base [(`f09f8af`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`0704a08`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/0704a08a04ed4d55d536c4b930bdf16eeb6bcd97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 285 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/1932?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...k/src/main/scala/org/apache/comet/serde/maps.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/1932?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fserde%2Fmaps.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9zZXJkZS9tYXBzLnNjYWxh) | 68.00% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/1932?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##               main    #1932       +/-   ##
   =============================================
   - Coverage     56.12%   33.60%   -22.52%     
   + Complexity      976      803      -173     
   =============================================
     Files           119      131       +12     
     Lines         11743    12872     +1129     
     Branches       2251     2390      +139     
   =============================================
   - Hits           6591     4326     -2265     
   - Misses         4012     7613     +3601     
   + Partials       1140      933      -207     
   ```
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/1932?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Feat: support map_from_arrays [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #1932:
URL: https://github.com/apache/datafusion-comet/pull/1932


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org