You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "tshauck (via GitHub)" <gi...@apache.org> on 2024/04/29 04:18:02 UTC
[PR] feat: ic for native unhex [datafusion-comet]
tshauck opened a new pull request, #342:
URL: https://github.com/apache/datafusion-comet/pull/342
## Which issue does this PR close?
Closes #341
## Rationale for this change
`unhex` is currently unsupported by comet
## What changes are included in this PR?
Add `unhex` as well as make some minor refactors.
## How are these changes tested?
Added simple tests to the rust and spark sql side of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1588760311
##########
spark/src/main/spark-3.2/org/apache/comet/shims/ShimCometUnhexExpr.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.comet.shims
+
+import org.apache.spark.sql.catalyst.expressions._
+
+/**
+ * `ShimCometExpr` parses the `Unhex` expression assuming that the catalyst version is 3.2.x.
Review Comment:
I think this trait will put all expr shims here. So maybe you can move this comment to `unhexSerde`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1593350009
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,25 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ val table = "unhex_table"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+
+ sql(s"""INSERT INTO $table VALUES
+ |('537061726B2053514C'),
+ |('737472696E67'),
+ |('\0'),
+ |(''),
+ |('###'),
+ |('G123'),
+ |('hello'),
+ |('A1B'),
+ |('0A1B')""".stripMargin)
+ checkSparkAnswerAndOperator(s"SELECT unhex(col) FROM $table")
Review Comment:
Made an update in https://github.com/apache/datafusion-comet/pull/342/commits/36baf8e87dd2689baf8c45961c7ed34a97f1f493 that seems to work (https://github.com/tshauck/arrow-datafusion-comet/actions/runs/8994978806). Happy to get any feedback about the approach taken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#issuecomment-2100675232
@viirya @kazuyukitanimura do you have any additional feedback?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589056095
##########
spark/src/main/spark-3.2/org/apache/comet/shims/ShimCometUnhexExpr.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.comet.shims
+
+import org.apache.spark.sql.catalyst.expressions._
+
+/**
+ * `ShimCometExpr` parses the `Unhex` expression assuming that the catalyst version is 3.2.x.
+ */
+trait ShimCometExpr {
Review Comment:
Maybe the name could be `CometExprShim` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1593233984
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,25 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ val table = "unhex_table"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+
+ sql(s"""INSERT INTO $table VALUES
+ |('537061726B2053514C'),
+ |('737472696E67'),
+ |('\0'),
+ |(''),
+ |('###'),
+ |('G123'),
+ |('hello'),
+ |('A1B'),
+ |('0A1B')""".stripMargin)
+ checkSparkAnswerAndOperator(s"SELECT unhex(col) FROM $table")
Review Comment:
Thanks, yeah, I looked at that bit. I think the test results are sorted, but the Spark case (left column) 27 sorts after 115.
Unfortunately, I think it may be legitimately different behavior between the 3.2 and the more recent versions.
<img width="995" alt="image" src="https://github.com/apache/datafusion-comet/assets/421839/36c59275-83b9-4099-9692-9913313f85c7"> .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#issuecomment-2094451349
Err... looks to be an issue w/ spark 3.2 I'll need to look into. Hopefully the majority of the code'll remain unchanged.
<img width="1280" alt="image" src="https://github.com/apache/datafusion-comet/assets/421839/0ea2d7a2-0122-48a9-bb87-bb384d22e7ef">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1593255210
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,25 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ val table = "unhex_table"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+
+ sql(s"""INSERT INTO $table VALUES
+ |('537061726B2053514C'),
+ |('737472696E67'),
+ |('\0'),
+ |(''),
+ |('###'),
+ |('G123'),
+ |('hello'),
+ |('A1B'),
+ |('0A1B')""".stripMargin)
+ checkSparkAnswerAndOperator(s"SELECT unhex(col) FROM $table")
Review Comment:
Actually adding the `ORDER BY` causes the test to fail because it no longer runs fully native, so ignore that suggestion.
Also, I do see code differences in Spark between 3.2 and 3.4 in the unhex algorithm. 3.2 does not have the `oddShift` variable. `oddShift` was added in https://github.com/apache/spark/commit/276abe3c3311300cffc6570b66f3977ea8172ff0
I guess the options are:
- mark `unhex` as incompat just for 3.2 and skip the test for 3.2 (probably the easiest path)
- implement per-spark-version logic in Rust for unhex
Let me know what you think or if you have any questions on this. This is a good example of the challenge of supporting multiple Spark versions 😓
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1594097078
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,27 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ assume(!isSpark32, "unhex function has incorrect behavior in 3.2")
Review Comment:
I am filing a follow on issue to document this incompatibility but it depends on some documentation infrastructure changes that I am working on. I will file a link to the issue here later today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589064578
##########
common/pom.xml:
##########
@@ -179,7 +179,8 @@ under the License.
</goals>
<configuration>
<sources>
- <source>src/main/${shims.source}</source>
+ <source>src/main/${shims.majorSource}</source>
+ <source>src/main/${shims.minorSource}</source>
Review Comment:
Thanks for adding the minor version shims. These are going to help me with some of work around supporting `cast`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589294755
##########
pom.xml:
##########
@@ -88,7 +88,8 @@ under the License.
<argLine>-ea -Xmx4g -Xss4m ${extraJavaTestArgs}</argLine>
<additional.3_3.test.source>spark-3.3-plus</additional.3_3.test.source>
<additional.3_4.test.source>spark-3.4</additional.3_4.test.source>
- <shims.source>spark-3.x</shims.source>
+ <shims.majorSource>spark-3.x</shims.majorSource>
+ <shims.minorSource>spark-3.4</shims.minorSource>
Review Comment:
How about `majorVerSrc` and `minorVerSrc`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1593213997
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,25 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ val table = "unhex_table"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+
+ sql(s"""INSERT INTO $table VALUES
+ |('537061726B2053514C'),
+ |('737472696E67'),
+ |('\0'),
+ |(''),
+ |('###'),
+ |('G123'),
+ |('hello'),
+ |('A1B'),
+ |('0A1B')""".stripMargin)
+ checkSparkAnswerAndOperator(s"SELECT unhex(col) FROM $table")
Review Comment:
It may be worth adding an `ORDER BY` clause here to ensure that the test is deterministic.
```suggestion
checkSparkAnswerAndOperator(s"SELECT unhex(col) FROM $table ORDER BY col")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1588843008
##########
core/src/execution/datafusion/expressions/scalar_funcs/unhex.rs:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::{Array, OffsetSizeTrait};
+use arrow_schema::DataType;
+use datafusion::logical_expr::ColumnarValue;
+use datafusion_common::{cast::as_generic_string_array, exec_err, DataFusionError, ScalarValue};
+
+fn unhex(string: &str, result: &mut Vec<u8>) -> Result<(), DataFusionError> {
+ if string.is_empty() {
+ return Ok(());
+ }
+
+ // Adjust the string if it has an odd length, and prepare to add a padding byte if needed.
+ let needs_padding = string.len() % 2 != 0;
+ let adjusted_string = if needs_padding { &string[1..] } else { string };
Review Comment:
If I understand this correctly, `string[0]` is discarded when the length is odd, is it intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589061362
##########
core/src/execution/datafusion/expressions/scalar_funcs/unhex.rs:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::{Array, OffsetSizeTrait};
+use arrow_schema::DataType;
+use datafusion::logical_expr::ColumnarValue;
+use datafusion_common::{cast::as_generic_string_array, exec_err, DataFusionError, ScalarValue};
+
+fn unhex(string: &str, result: &mut Vec<u8>) -> Result<(), DataFusionError> {
+ if string.is_empty() {
+ return Ok(());
+ }
+
+ // Adjust the string if it has an odd length, and prepare to add a padding byte if needed.
+ let needs_padding = string.len() % 2 != 0;
+ let adjusted_string = if needs_padding { &string[1..] } else { string };
Review Comment:
Here is the logic in Spark 3.4.2 for handling the first char if the input is padded, for reference. It looks like there is some validation of the first digit that we do not have in this PR and it also looks like the unhexed digit is stored in the output is used in the return value if the length of the input string is 1. It would be good to make sure that we have tests covering this case.
```scala
if ((bytes.length & 0x01) != 0) {
// padding with '0'
if (bytes(0) < 0) {
return null
}
val v = Hex.unhexDigits(bytes(0))
if (v == -1) {
return null
}
out(0) = v
i += 1
oddShift = 1
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1584002088
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1396,6 +1396,17 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde {
val optExpr = scalarExprToProto("atan2", leftExpr, rightExpr)
optExprWithInfo(optExpr, expr, left, right)
+ case e @ Unhex(child, failOnError) =>
Review Comment:
I was wrong, actually `getFailOnError` needs to be modified for this approach as `unhex` is `unaryExpression`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1582524648
##########
core/src/execution/datafusion/planner.rs:
##########
@@ -1301,24 +1301,26 @@ impl PhysicalPlanner {
.iter()
.map(|x| x.data_type(input_schema.as_ref()))
.collect::<Result<Vec<_>, _>>()?;
+
let data_type = match expr.return_type.as_ref().map(to_arrow_datatype) {
Some(t) => t,
None => {
// If no data type is provided from Spark, we'll use DF's return type from the
// scalar function
// Note this assumes the `fun_name` is a defined function in DF. Otherwise, it'll
// throw error.
- let fun = BuiltinScalarFunction::from_str(fun_name);
- if fun.is_err() {
+
+ if let Ok(fun) = BuiltinScalarFunction::from_str(fun_name) {
Review Comment:
Unrelated, but more idiomatic IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1593221554
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,25 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ val table = "unhex_table"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+
+ sql(s"""INSERT INTO $table VALUES
+ |('537061726B2053514C'),
+ |('737472696E67'),
+ |('\0'),
+ |(''),
+ |('###'),
+ |('G123'),
+ |('hello'),
+ |('A1B'),
+ |('0A1B')""".stripMargin)
+ checkSparkAnswerAndOperator(s"SELECT unhex(col) FROM $table")
Review Comment:
I wonder if the Spark 3.2 failure is at all related to ordering?
```
[WrappedArray(10, 27)] [WrappedArray(10, 27)]
![WrappedArray(115, 116, 114, 105, 110, 103)] [WrappedArray(10, 27)]
![WrappedArray(27, 0)] [WrappedArray(115, 116, 114, 105, 110, 103)]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1588759861
##########
spark/src/main/spark-3.2/org/apache/comet/shims/ShimCometUnhexExpr.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.comet.shims
+
+import org.apache.spark.sql.catalyst.expressions._
+
+/**
+ * `ShimCometExpr` parses the `Unhex` expression assuming that the catalyst version is 3.2.x.
+ */
+trait ShimCometExpr {
Review Comment:
The file name should be `ShimCometExpr.scala` instead of `ShimCometUnhexExpr.scala`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1593238627
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,25 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ val table = "unhex_table"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+
+ sql(s"""INSERT INTO $table VALUES
+ |('537061726B2053514C'),
+ |('737472696E67'),
+ |('\0'),
+ |(''),
+ |('###'),
+ |('G123'),
+ |('hello'),
+ |('A1B'),
+ |('0A1B')""".stripMargin)
+ checkSparkAnswerAndOperator(s"SELECT unhex(col) FROM $table")
Review Comment:
Looking at the spark repo a bit more, it looks like there was a bug in the [3.2 implementation](https://github.com/apache/spark/commit/276abe3c3311300cffc6570b66f3977ea8172ff0) that was fixed in subsequent versions.
How should I handle this? Like should I write a native implementation of the erroneous code in order to be faithful to the spark, or have it be correct if different?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1593261305
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,25 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ val table = "unhex_table"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+
+ sql(s"""INSERT INTO $table VALUES
+ |('537061726B2053514C'),
+ |('737472696E67'),
+ |('\0'),
+ |(''),
+ |('###'),
+ |('G123'),
+ |('hello'),
+ |('A1B'),
+ |('0A1B')""".stripMargin)
+ checkSparkAnswerAndOperator(s"SELECT unhex(col) FROM $table")
Review Comment:
Thanks for looking, I think we found the same commit w.r.t. 3.2's differences.
I'll just go the easy route since matching 3.2 would be implementing buggy code and 3.2 is the oldest supported version (so presumably it'll be dropped at some point). I'll use this as a chance to poke around the repo, but may ask in discord if I get blocked on marking it incompat for 3.2 and skipping the tests for that version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1588760311
##########
spark/src/main/spark-3.2/org/apache/comet/shims/ShimCometUnhexExpr.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.comet.shims
+
+import org.apache.spark.sql.catalyst.expressions._
+
+/**
+ * `ShimCometExpr` parses the `Unhex` expression assuming that the catalyst version is 3.2.x.
Review Comment:
I think this trait will put all expr shims here. So maybe you can move this comment to `unhexSerde` as it is for `Unhex` expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589056857
##########
core/src/execution/datafusion/expressions/scalar_funcs/unhex.rs:
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::{Array, OffsetSizeTrait};
+use arrow_schema::DataType;
+use datafusion::logical_expr::ColumnarValue;
+use datafusion_common::{cast::as_generic_string_array, exec_err, DataFusionError, ScalarValue};
+
+fn unhex(string: &str, result: &mut Vec<u8>) -> Result<(), DataFusionError> {
+ if string.is_empty() {
+ return Ok(());
+ }
+
+ // Adjust the string if it has an odd length, and prepare to add a padding byte if needed.
+ let needs_padding = string.len() % 2 != 0;
+ let adjusted_string = if needs_padding { &string[1..] } else { string };
+
+ let mut iter = adjusted_string.chars().peekable();
+ while let (Some(high_char), Some(low_char)) = (iter.next(), iter.next()) {
+ let high = high_char
+ .to_digit(16)
+ .ok_or_else(|| DataFusionError::Internal("Invalid hex character".to_string()))?;
+ let low = low_char
+ .to_digit(16)
+ .ok_or_else(|| DataFusionError::Internal("Invalid hex character".to_string()))?;
+
+ result.push((high << 4 | low) as u8);
+ }
+
+ if needs_padding {
+ result.push(0);
+ }
+
+ Ok(())
+}
+
+fn spark_unhex_inner<T: OffsetSizeTrait>(
+ array: &ColumnarValue,
+ fail_on_error: bool,
+) -> Result<ColumnarValue, DataFusionError> {
+ match array {
+ ColumnarValue::Array(array) => {
+ let string_array = as_generic_string_array::<T>(array)?;
+
+ let mut builder = arrow::array::BinaryBuilder::new();
+ let mut encoded = Vec::new();
+
+ for i in 0..string_array.len() {
+ let string = string_array.value(i);
+
+ if unhex(string, &mut encoded).is_ok() {
+ builder.append_value(encoded.as_slice());
+ encoded.clear();
+ } else if fail_on_error {
+ return exec_err!("Input to unhex is not a valid hex string: {string}");
+ } else {
+ builder.append_null();
+ }
+ }
+ Ok(ColumnarValue::Array(Arc::new(builder.finish())))
+ }
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(string))) => {
+ let mut encoded = Vec::new();
+
+ if unhex(string, &mut encoded).is_ok() {
+ Ok(ColumnarValue::Scalar(ScalarValue::Binary(Some(encoded))))
+ } else if fail_on_error {
+ exec_err!("Input to unhex is not a valid hex string: {string}")
+ } else {
+ Ok(ColumnarValue::Scalar(ScalarValue::Binary(None)))
+ }
+ }
+ _ => {
+ exec_err!(
+ "The first argument must be a string scalar or array, but got: {:?}",
+ array
+ )
+ }
+ }
+}
+
+pub(super) fn spark_unhex(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
+ if args.len() > 2 {
+ return exec_err!("unhex takes at most 2 arguments, but got: {}", args.len());
+ }
+
+ let val_to_unhex = &args[0];
+ let fail_on_error = if args.len() == 2 {
+ match &args[1] {
+ ColumnarValue::Scalar(ScalarValue::Boolean(Some(fail_on_error))) => *fail_on_error,
+ _ => {
+ return exec_err!(
+ "The second argument must be boolean scalar, but got: {:?}",
+ args[1]
+ );
+ }
+ }
+ } else {
+ false
+ };
+
+ match val_to_unhex.data_type() {
+ DataType::Utf8 => spark_unhex_inner::<i32>(val_to_unhex, fail_on_error),
+ DataType::LargeUtf8 => spark_unhex_inner::<i64>(val_to_unhex, fail_on_error),
+ other => exec_err!(
+ "The first argument must be a string scalar or array, but got: {:?}",
+ other
+ ),
+ }
+}
+
+#[cfg(test)]
+mod test {
+ use super::unhex;
+
+ #[test]
+ fn test_unhex() -> Result<(), Box<dyn std::error::Error>> {
+ let mut result = Vec::new();
+
+ unhex("537061726B2053514C", &mut result)?;
Review Comment:
Could we also have a test for the case where the input is padded?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589662961
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1025,6 +1025,19 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
+ test("unhex") {
+ Seq(false, true).foreach { dictionary =>
+ withSQLConf("parquet.enable.dictionary" -> dictionary.toString) {
Review Comment:
This was somewhat blind copying of other stuff. I've removed it in [112c7c6](https://github.com/apache/datafusion-comet/pull/342/commits/112c7c61db86f9a7383b0667af846ac02b1703c7). There's some other feedback around test breadth, so just simplifying the test for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1585820645
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1396,6 +1396,17 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde {
val optExpr = scalarExprToProto("atan2", leftExpr, rightExpr)
optExprWithInfo(optExpr, expr, left, right)
+ case e @ Unhex(child, failOnError) =>
Review Comment:
@andygrove Would you mind having a look at the updated PR and let me know what you think of the shim implemntation? The thought was to setup a major and minor shim to keep the existing shim for all of spark 3 then apply the more granular difference in the minor. It gets a little weird in that I'm applying 3.3's shim to 3.2.
@kazuyukitanimura Thanks for looking!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1594127098
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,27 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ assume(!isSpark32, "unhex function has incorrect behavior in 3.2")
Review Comment:
follow up issue: https://github.com/apache/datafusion-comet/issues/400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589664005
##########
spark/src/main/spark-3.2/org/apache/comet/shims/ShimCometUnhexExpr.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.comet.shims
+
+import org.apache.spark.sql.catalyst.expressions._
+
+/**
+ * `ShimCometExpr` parses the `Unhex` expression assuming that the catalyst version is 3.2.x.
+ */
+trait ShimCometExpr {
Review Comment:
Thanks for the feedback, updated to `CometExprShim` in https://github.com/apache/datafusion-comet/pull/342/commits/a378f7433836d81543c4e6b8ce6c108934a893c5 along with updating the docstrings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1588762861
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1025,6 +1025,19 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
+ test("unhex") {
+ Seq(false, true).foreach { dictionary =>
+ withSQLConf("parquet.enable.dictionary" -> dictionary.toString) {
Review Comment:
Hmm, as there is only one value, does the `dictionary` actually work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1582524793
##########
core/src/execution/datafusion/expressions/scalar_funcs.rs:
##########
@@ -123,11 +126,10 @@ pub fn create_comet_physical_fun(
make_comet_scalar_udf!(spark_func_name, wrapped_func, without data_type)
}
_ => {
- let fun = BuiltinScalarFunction::from_str(fun_name);
- if fun.is_err() {
- Ok(ScalarFunctionDefinition::UDF(registry.udf(fun_name)?))
+ if let Ok(fun) = BuiltinScalarFunction::from_str(fun_name) {
Review Comment:
Unrelated, but more idiomatic IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589290483
##########
spark/src/main/spark-3.2/org/apache/comet/shims/ShimCometUnhexExpr.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.comet.shims
+
+import org.apache.spark.sql.catalyst.expressions._
+
+/**
+ * `ShimCometExpr` parses the `Unhex` expression assuming that the catalyst version is 3.2.x.
+ */
+trait ShimCometExpr {
Review Comment:
+1 for `CometExprShim`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#issuecomment-2094372353
I think this is ready for review. I updated the `unhex` impl to be more faithful to Spark's (for odd-length inputs in particular), added better null handling, and added more tests from the Spark repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589664353
##########
spark/src/main/spark-3.2/org/apache/comet/shims/ShimCometUnhexExpr.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.comet.shims
+
+import org.apache.spark.sql.catalyst.expressions._
+
+/**
+ * `ShimCometExpr` parses the `Unhex` expression assuming that the catalyst version is 3.2.x.
Review Comment:
Thanks, updated in https://github.com/apache/datafusion-comet/pull/342/commits/a378f7433836d81543c4e6b8ce6c108934a893c5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1583998358
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1396,6 +1396,17 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde {
val optExpr = scalarExprToProto("atan2", leftExpr, rightExpr)
optExprWithInfo(optExpr, expr, left, right)
+ case e @ Unhex(child, failOnError) =>
Review Comment:
I think we can handle this using existing shim, something like
```
case e @ Unhex =>
e.child...
getFailOnError(e) ...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1585982628
##########
pom.xml:
##########
@@ -88,7 +88,8 @@ under the License.
<argLine>-ea -Xmx4g -Xss4m ${extraJavaTestArgs}</argLine>
<additional.3_3.test.source>spark-3.3-plus</additional.3_3.test.source>
<additional.3_4.test.source>spark-3.4</additional.3_4.test.source>
- <shims.source>spark-3.x</shims.source>
+ <shims.majorSource>spark-3.x</shims.majorSource>
+ <shims.minorSource>spark-3.4.x</shims.minorSource>
Review Comment:
FYI `spark-3.x` will be gone and supposed to be replaced by independent `spark-3.2`, `spark-3.3`, and `spark-3.4` dirs
So let's go ahead and start using `spark-3.4` instead of `spark-3.4.x`? I.e.
```
<shims.source.shared>spark-3.x</shims.source.shared>
<shims.source>spark-3.4</shims.source>
```
##########
pom.xml:
##########
@@ -500,6 +501,7 @@ under the License.
<!-- we don't add special test suits for spark-3.2, so a not existed dir is specified-->
<additional.3_3.test.source>not-needed-yet</additional.3_3.test.source>
<additional.3_4.test.source>not-needed-yet</additional.3_4.test.source>
+ <shims.minorSource>spark-3.3.x</shims.minorSource>
Review Comment:
I guess this should be `spark-3.2`?
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1396,6 +1397,19 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde {
val optExpr = scalarExprToProto("atan2", leftExpr, rightExpr)
optExprWithInfo(optExpr, expr, left, right)
+ case e: Unhex =>
+ val unHex = unhexSerde(e)
+
+ val childCast = Cast(unHex._1, StringType)
+ val failOnErrorCast = Cast(unHex._2, BooleanType)
Review Comment:
What would happen if we do not use `Cast`?
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -45,12 +45,13 @@ import org.apache.comet.CometSparkSessionExtensions.{isCometOperatorEnabled, isC
import org.apache.comet.serde.ExprOuterClass.{AggExpr, DataType => ProtoDataType, Expr, ScalarFunc}
import org.apache.comet.serde.ExprOuterClass.DataType.{DataTypeInfo, DecimalInfo, ListInfo, MapInfo, StructInfo}
import org.apache.comet.serde.OperatorOuterClass.{AggregateMode => CometAggregateMode, JoinType, Operator}
+import org.apache.comet.shims.ShimCometUnhexExpr
import org.apache.comet.shims.ShimQueryPlanSerde
/**
* An utility object for query plan and expression serialization.
*/
-object QueryPlanSerde extends Logging with ShimQueryPlanSerde {
+object QueryPlanSerde extends Logging with ShimQueryPlanSerde with ShimCometUnhexExpr {
Review Comment:
I would say `ShimCometUnhexExpr` should be more generic name, not unhex specific. What about `ShimCometExpr ` (or if you have other idea, please feel free to propose)?
Otherwise, we will have to keep adding trait class per function.
Eventually we should merge `ShimQueryPlanSerde` and `ShimCometExpr` into one when we remove `spark-3.x` dir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1594094594
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,27 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ assume(!isSpark32, "unhex function has incorrect behavior in 3.2")
+ val table = "unhex_table"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+
+ sql(s"""INSERT INTO $table VALUES
+ |('537061726B2053514C'),
+ |('737472696E67'),
+ |('\\0'),
Review Comment:
Removing this test input fixes the Scala 2.13 build errors. I confirmed locally with `mvn clean package -DskipTests -Pscala-2.13`.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#issuecomment-2101377549
I plan on merging this tomorrow if there is no more feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1582536029
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1396,6 +1396,17 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde {
val optExpr = scalarExprToProto("atan2", leftExpr, rightExpr)
optExprWithInfo(optExpr, expr, left, right)
+ case e @ Unhex(child, failOnError) =>
Review Comment:
https://github.com/apache/spark/blob/59d5946cfd377e9203ccf572deb34f87fab7510c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L1152
vs
https://github.com/apache/spark/blob/45ba9224602eb18fe45e339cbb8cf2e8a4924f0b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L1125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1594082373
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,27 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ assume(!isSpark32, "unhex function has incorrect behavior in 3.2")
Review Comment:
```suggestion
// When running against Spark 3.2, we include a bug fix for https://issues.apache.org/jira/browse/SPARK-40924 that
// was added in Spark 3.3, so although Comet's behavior is more correct when running against Spark 3.2, it is not
// the same (and this only applies to edge cases with hex inputs with lengths that are not divisible by 2)
assume(!isSpark32, "unhex function has incorrect behavior in 3.2")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1594773167
##########
core/src/execution/datafusion/expressions/scalar_funcs/unhex.rs:
##########
@@ -0,0 +1,229 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::OffsetSizeTrait;
+use arrow_schema::DataType;
+use datafusion::logical_expr::ColumnarValue;
+use datafusion_common::{cast::as_generic_string_array, exec_err, DataFusionError, ScalarValue};
+
+/// Helper function to convert a hex digit to a binary value.
+fn unhex_digit(c: u8) -> Result<u8, DataFusionError> {
+ match c {
+ b'0'..=b'9' => Ok(c - b'0'),
+ b'A'..=b'F' => Ok(10 + c - b'A'),
+ b'a'..=b'f' => Ok(10 + c - b'a'),
+ _ => Err(DataFusionError::Execution(
+ "Input to unhex_digit is not a valid hex digit".to_string(),
+ )),
+ }
+}
+
+/// Convert a hex string to binary and store the result in `result`. Returns an error if the input
+/// is not a valid hex string.
+fn unhex(hex_str: &str, result: &mut Vec<u8>) -> Result<(), DataFusionError> {
+ let bytes = hex_str.as_bytes();
+
+ let mut i = 0;
+
+ if (bytes.len() & 0x01) != 0 {
+ let v = unhex_digit(bytes[0])?;
+
+ result.push(v);
+ i += 1;
+ }
+
+ while i < bytes.len() {
+ let first = unhex_digit(bytes[i])?;
+ let second = unhex_digit(bytes[i + 1])?;
+ result.push((first << 4) | second);
+
+ i += 2;
+ }
+
+ Ok(())
+}
+
+fn spark_unhex_inner<T: OffsetSizeTrait>(
+ array: &ColumnarValue,
+ fail_on_error: bool,
+) -> Result<ColumnarValue, DataFusionError> {
+ match array {
+ ColumnarValue::Array(array) => {
+ let string_array = as_generic_string_array::<T>(array)?;
+
+ let mut encoded = Vec::new();
+ let mut builder = arrow::array::BinaryBuilder::new();
+
+ for item in string_array.iter() {
+ if let Some(s) = item {
+ if unhex(s, &mut encoded).is_ok() {
+ builder.append_value(encoded.as_slice());
+ encoded.clear();
+ } else if fail_on_error {
+ return exec_err!("Input to unhex is not a valid hex string: {s}");
+ } else {
+ builder.append_null();
Review Comment:
All good, thank you very much for looking. I think you're right in that there was a bug, which should be fixed and tested in https://github.com/apache/datafusion-comet/pull/342/commits/c5c3fcd51edddba44d1dd3b5ad176066ece0a5eb.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#issuecomment-2092243886
Hi, I updated the shim handling for 3.2 and made various other updates (based on PR feedback and general cleanup). Please have another look and let me know what you think, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#issuecomment-2098985033
Thanks for the updates @tshauck. I plan on reviewing later today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1585863854
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1396,6 +1396,17 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde {
val optExpr = scalarExprToProto("atan2", leftExpr, rightExpr)
optExprWithInfo(optExpr, expr, left, right)
+ case e @ Unhex(child, failOnError) =>
Review Comment:
Thanks @tshauck I will review this tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589662280
##########
pom.xml:
##########
@@ -88,7 +88,8 @@ under the License.
<argLine>-ea -Xmx4g -Xss4m ${extraJavaTestArgs}</argLine>
<additional.3_3.test.source>spark-3.3-plus</additional.3_3.test.source>
<additional.3_4.test.source>spark-3.4</additional.3_4.test.source>
- <shims.source>spark-3.x</shims.source>
+ <shims.majorSource>spark-3.x</shims.majorSource>
+ <shims.minorSource>spark-3.4</shims.minorSource>
Review Comment:
Fine by me, done in [112c7c6](https://github.com/apache/datafusion-comet/pull/342/commits/112c7c61db86f9a7383b0667af846ac02b1703c7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#issuecomment-2132500112
I forgot to ask about dictionary and scalar. Filed https://github.com/apache/datafusion-comet/issues/477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1593238627
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,25 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ val table = "unhex_table"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+
+ sql(s"""INSERT INTO $table VALUES
+ |('537061726B2053514C'),
+ |('737472696E67'),
+ |('\0'),
+ |(''),
+ |('###'),
+ |('G123'),
+ |('hello'),
+ |('A1B'),
+ |('0A1B')""".stripMargin)
+ checkSparkAnswerAndOperator(s"SELECT unhex(col) FROM $table")
Review Comment:
Looking at the spark repo a bit more, it looks like there was a bug in the [3.2 implementation](https://github.com/apache/spark/commit/276abe3c3311300cffc6570b66f3977ea8172ff0) that was fixed in subsequent versions.
How should I handle this? Like should I write a native implementation of the erroneous code in order to be faithful to the spark, have it be correct if different, or maybe just fallback to the spark implementation on that version (not sure if that's possible)? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1594749074
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1396,6 +1397,16 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde {
val optExpr = scalarExprToProto("atan2", leftExpr, rightExpr)
optExprWithInfo(optExpr, expr, left, right)
+ case e: Unhex if !isSpark32 =>
Review Comment:
Unrelated, but potentially we can vote in the community when to deprecate 3.2 support...
##########
core/src/execution/datafusion/expressions/scalar_funcs/unhex.rs:
##########
@@ -0,0 +1,229 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::OffsetSizeTrait;
+use arrow_schema::DataType;
+use datafusion::logical_expr::ColumnarValue;
+use datafusion_common::{cast::as_generic_string_array, exec_err, DataFusionError, ScalarValue};
+
+/// Helper function to convert a hex digit to a binary value.
+fn unhex_digit(c: u8) -> Result<u8, DataFusionError> {
+ match c {
+ b'0'..=b'9' => Ok(c - b'0'),
+ b'A'..=b'F' => Ok(10 + c - b'A'),
+ b'a'..=b'f' => Ok(10 + c - b'a'),
+ _ => Err(DataFusionError::Execution(
+ "Input to unhex_digit is not a valid hex digit".to_string(),
+ )),
+ }
+}
+
+/// Convert a hex string to binary and store the result in `result`. Returns an error if the input
+/// is not a valid hex string.
+fn unhex(hex_str: &str, result: &mut Vec<u8>) -> Result<(), DataFusionError> {
+ let bytes = hex_str.as_bytes();
+
+ let mut i = 0;
+
+ if (bytes.len() & 0x01) != 0 {
+ let v = unhex_digit(bytes[0])?;
+
+ result.push(v);
+ i += 1;
+ }
+
+ while i < bytes.len() {
+ let first = unhex_digit(bytes[i])?;
+ let second = unhex_digit(bytes[i + 1])?;
+ result.push((first << 4) | second);
+
+ i += 2;
+ }
+
+ Ok(())
+}
+
+fn spark_unhex_inner<T: OffsetSizeTrait>(
+ array: &ColumnarValue,
+ fail_on_error: bool,
+) -> Result<ColumnarValue, DataFusionError> {
+ match array {
+ ColumnarValue::Array(array) => {
+ let string_array = as_generic_string_array::<T>(array)?;
+
+ let mut encoded = Vec::new();
+ let mut builder = arrow::array::BinaryBuilder::new();
+
+ for item in string_array.iter() {
+ if let Some(s) = item {
+ if unhex(s, &mut encoded).is_ok() {
+ builder.append_value(encoded.as_slice());
+ encoded.clear();
+ } else if fail_on_error {
+ return exec_err!("Input to unhex is not a valid hex string: {s}");
+ } else {
+ builder.append_null();
Review Comment:
If `unhex()` fails and `fail_on_error=false` encoded is not cleared? Is this a same behavior as Spark? It would be great if we could add a test to verify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1594744119
##########
core/src/execution/datafusion/expressions/scalar_funcs/unhex.rs:
##########
@@ -0,0 +1,229 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::OffsetSizeTrait;
+use arrow_schema::DataType;
+use datafusion::logical_expr::ColumnarValue;
+use datafusion_common::{cast::as_generic_string_array, exec_err, DataFusionError, ScalarValue};
+
+/// Helper function to convert a hex digit to a binary value.
+fn unhex_digit(c: u8) -> Result<u8, DataFusionError> {
+ match c {
+ b'0'..=b'9' => Ok(c - b'0'),
+ b'A'..=b'F' => Ok(10 + c - b'A'),
+ b'a'..=b'f' => Ok(10 + c - b'a'),
+ _ => Err(DataFusionError::Execution(
+ "Input to unhex_digit is not a valid hex digit".to_string(),
+ )),
+ }
+}
+
+/// Convert a hex string to binary and store the result in `result`. Returns an error if the input
+/// is not a valid hex string.
+fn unhex(hex_str: &str, result: &mut Vec<u8>) -> Result<(), DataFusionError> {
+ let bytes = hex_str.as_bytes();
+
+ let mut i = 0;
+
+ if (bytes.len() & 0x01) != 0 {
+ let v = unhex_digit(bytes[0])?;
+
+ result.push(v);
+ i += 1;
+ }
+
+ while i < bytes.len() {
+ let first = unhex_digit(bytes[i])?;
+ let second = unhex_digit(bytes[i + 1])?;
+ result.push((first << 4) | second);
+
+ i += 2;
+ }
+
+ Ok(())
+}
+
+fn spark_unhex_inner<T: OffsetSizeTrait>(
+ array: &ColumnarValue,
+ fail_on_error: bool,
+) -> Result<ColumnarValue, DataFusionError> {
+ match array {
+ ColumnarValue::Array(array) => {
+ let string_array = as_generic_string_array::<T>(array)?;
+
+ let mut encoded = Vec::new();
+ let mut builder = arrow::array::BinaryBuilder::new();
+
+ for item in string_array.iter() {
+ if let Some(s) = item {
+ if unhex(s, &mut encoded).is_ok() {
+ builder.append_value(encoded.as_slice());
+ encoded.clear();
+ } else if fail_on_error {
+ return exec_err!("Input to unhex is not a valid hex string: {s}");
+ } else {
+ builder.append_null();
Review Comment:
If `unhex()` fails and `fail_on_error=false`, the `encoded` is not cleared? Is this a same behavior as Spark? It would be great if we could add a test to verify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1594094594
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1024,7 +1024,27 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+ test("unhex") {
+ assume(!isSpark32, "unhex function has incorrect behavior in 3.2")
+ val table = "unhex_table"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+
+ sql(s"""INSERT INTO $table VALUES
+ |('537061726B2053514C'),
+ |('737472696E67'),
+ |('\\0'),
Review Comment:
Removing this test fixes the Scala 2.13 build errors. I confirmed locally with `mvn clean package -DskipTests -Pscala-2.13`.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1588761968
##########
pom.xml:
##########
@@ -88,7 +88,8 @@ under the License.
<argLine>-ea -Xmx4g -Xss4m ${extraJavaTestArgs}</argLine>
<additional.3_3.test.source>spark-3.3-plus</additional.3_3.test.source>
<additional.3_4.test.source>spark-3.4</additional.3_4.test.source>
- <shims.source>spark-3.x</shims.source>
+ <shims.majorSource>spark-3.x</shims.majorSource>
+ <shims.minorSource>spark-3.4</shims.minorSource>
Review Comment:
Hmm, I don't get the naming? What `majorSource` and `minorSource` mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1589063630
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1025,6 +1025,19 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
+ test("unhex") {
+ Seq(false, true).foreach { dictionary =>
+ withSQLConf("parquet.enable.dictionary" -> dictionary.toString) {
+ val table = "test"
+ withTable(table) {
+ sql(s"create table $table(col string) using parquet")
+ sql(s"insert into $table values('537061726B2053514C')")
Review Comment:
I'd like to see more values being tested here, both valid and invalid, and covering the padded case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1586005147
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1396,6 +1396,17 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde {
val optExpr = scalarExprToProto("atan2", leftExpr, rightExpr)
optExprWithInfo(optExpr, expr, left, right)
+ case e @ Unhex(child, failOnError) =>
Review Comment:
I think we can have duplicated shim for 3.3 and 3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: ic for native unhex [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#discussion_r1583437769
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1396,6 +1396,17 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde {
val optExpr = scalarExprToProto("atan2", leftExpr, rightExpr)
optExprWithInfo(optExpr, expr, left, right)
+ case e @ Unhex(child, failOnError) =>
Review Comment:
We need to use a shim approach. We currently have one folder with source for spark-3.x. We should add additional folders as needed e.g. spark-3.3.x / spark-3.4.x with any code that is specific to those versions.
I am happy to help with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement Spark unhex [datafusion-comet]
Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on PR #342:
URL: https://github.com/apache/datafusion-comet/pull/342#issuecomment-2093665208
Thanks for all the feedback. I think I've addressed the build/naming/etc feedback, and will have a look at improving the tests and any associated implementation changes sometime tomorrow. I'll request another review via GH when it's ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org