You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "KarpagamKarthikeyan (via GitHub)" <gi...@apache.org> on 2026/07/21 20:08:21 UTC
[PR] feat: add Spark-compatible hypot function [datafusion]
KarpagamKarthikeyan opened a new pull request, #23774:
URL: https://github.com/apache/datafusion/pull/23774
## Which issue does this PR close?
- Closes #23770
- Part of #15914
## Rationale for this change
Spark provides [`hypot(expr1, expr2)`](https://spark.apache.org/docs/latest/api/sql/#hypot), which returns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow or underflow. It was not yet implemented in `datafusion-spark` — only an auto-generated test stub existed at `spark/math/hypot.slt` with its query commented out.
## What changes are included in this PR?
- Add `SparkHypot` (implementing `ScalarUDFImpl`) in `datafusion/spark/src/function/math/hypot.rs`, backed by Rust's `f64::hypot` — the same overflow-safe algorithm as Java/Spark's `Math.hypot`.
- Register it in `datafusion/spark/src/function/math/mod.rs`.
- Enable the `hypot.slt` sqllogictest.
The signature is `exact(Float64, Float64) -> Float64`, following the `datafusion-spark` convention of only accepting types Spark supports. Computation uses the Arrow `binary` kernel so NULL in either argument propagates to a NULL result, matching Spark.
## Are these changes tested?
Yes — `datafusion/sqllogictest/test_files/spark/math/hypot.slt` covers:
- scalar Pythagorean triples (`hypot(3, 4)` → 5, `hypot(5, 12)` → 13),
- double inputs,
- NULL propagation when either argument is NULL,
- the array path (including a NULL row),
- overflow-safety: `hypot(3e200, 4e200)` stays finite, whereas a naive `sqrt(a^2 + b^2)` would overflow to `Infinity`.
## Are there any user-facing changes?
Yes — adds the Spark-compatible `hypot` scalar function to `datafusion-spark`. No breaking changes to public APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add Spark-compatible hypot function [datafusion]
Posted by "KarpagamKarthikeyan (via GitHub)" <gi...@apache.org>.
KarpagamKarthikeyan commented on code in PR #23774:
URL: https://github.com/apache/datafusion/pull/23774#discussion_r3649146624
##########
datafusion/spark/src/function/math/hypot.rs:
##########
@@ -0,0 +1,90 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, AsArray, Float64Array};
+use arrow::compute::kernels::arity::binary;
+use arrow::datatypes::{DataType, Float64Type};
+use datafusion_common::Result;
+use datafusion_common::utils::take_function_args;
+use datafusion_expr::{
+ ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+
+/// Spark-compatible `hypot` function.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#hypot>
+///
+/// Returns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow or
+/// underflow, matching Spark's use of `java.lang.Math.hypot`.
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkHypot {
+ signature: Signature,
+}
+
+impl Default for SparkHypot {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkHypot {
+ pub fn new() -> Self {
+ Self {
+ // Spark only defines hypot over doubles; `exact` makes coercion
+ // guarantee both inputs are Float64 before `invoke` runs.
+ signature: Signature::exact(
+ vec![DataType::Float64, DataType::Float64],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkHypot {
+ fn name(&self) -> &str {
+ "hypot"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
+ Ok(DataType::Float64)
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let num_rows = args.number_rows;
+ let [x, y] = take_function_args(self.name(), args.args)?;
+
+ // Broadcast scalars to arrays so one path covers every combination.
+ let x = x.to_array(num_rows)?;
Review Comment:
Thanks! Refactored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add Spark-compatible hypot function [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #23774:
URL: https://github.com/apache/datafusion/pull/23774
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add Spark-compatible hypot function [datafusion]
Posted by "KarpagamKarthikeyan (via GitHub)" <gi...@apache.org>.
KarpagamKarthikeyan commented on PR #23774:
URL: https://github.com/apache/datafusion/pull/23774#issuecomment-5038838389
cc @Jefffrey @andygrove — this is my first contribution, so CI is currently awaiting a committer's approval. Would really appreciate a trigger when you have a moment.
This adds a Spark-compatible `hypot` function to `datafusion-spark`, following the existing `math` function conventions (`exact(Float64, Float64) -> Float64`, `invoke` rather than `simplify`), backed by Rust's overflow-safe `f64::hypot`. Covered by sqllogictest cases (triples, doubles, NULL propagation, array path, and overflow-safety). Closes #23770.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add Spark-compatible hypot function [datafusion]
Posted by "KarpagamKarthikeyan (via GitHub)" <gi...@apache.org>.
KarpagamKarthikeyan commented on code in PR #23774:
URL: https://github.com/apache/datafusion/pull/23774#discussion_r3649147039
##########
datafusion/sqllogictest/test_files/spark/math/hypot.slt:
##########
@@ -21,7 +21,45 @@
# For more information, please see:
# https://github.com/apache/datafusion/issues/15914
-## Original Query: SELECT hypot(3, 4);
-## PySpark 3.5.5 Result: {'HYPOT(3, 4)': 5.0, 'typeof(HYPOT(3, 4))': 'double', 'typeof(3)': 'int', 'typeof(4)': 'int'}
-#query
-#SELECT hypot(3::int, 4::int);
+# Scalar: classic Pythagorean triples (3-4-5, 5-12-13)
Review Comment:
Done!
##########
datafusion/spark/src/function/math/hypot.rs:
##########
@@ -0,0 +1,90 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, AsArray, Float64Array};
+use arrow::compute::kernels::arity::binary;
+use arrow::datatypes::{DataType, Float64Type};
+use datafusion_common::Result;
+use datafusion_common::utils::take_function_args;
+use datafusion_expr::{
+ ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+
+/// Spark-compatible `hypot` function.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#hypot>
+///
+/// Returns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow or
+/// underflow, matching Spark's use of `java.lang.Math.hypot`.
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkHypot {
+ signature: Signature,
+}
+
+impl Default for SparkHypot {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkHypot {
+ pub fn new() -> Self {
+ Self {
+ // Spark only defines hypot over doubles; `exact` makes coercion
+ // guarantee both inputs are Float64 before `invoke` runs.
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add Spark-compatible hypot function [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23774:
URL: https://github.com/apache/datafusion/pull/23774#issuecomment-5053429791
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/23774?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `78.78788%` with `7 lines` in your changes missing coverage. Please review.
:warning: Please [upload](https://docs.codecov.com/docs/codecov-uploader) report for BASE (`main@05b7e11`). [Learn more](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#section-missing-base-commit) about missing BASE report.
:warning: Report is 6 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/23774?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/spark/src/function/math/hypot.rs](https://app.codecov.io/gh/apache/datafusion/pull/23774?src=pr&el=tree&filepath=datafusion%2Fspark%2Fsrc%2Ffunction%2Fmath%2Fhypot.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zcGFyay9zcmMvZnVuY3Rpb24vbWF0aC9oeXBvdC5ycw==) | 78.12% | [3 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23774?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #23774 +/- ##
=======================================
Coverage ? 80.71%
=======================================
Files ? 1090
Lines ? 368933
Branches ? 368933
=======================================
Hits ? 297789
Misses ? 53381
Partials ? 17763
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/23774?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add Spark-compatible hypot function [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #23774:
URL: https://github.com/apache/datafusion/pull/23774#discussion_r3634982335
##########
datafusion/spark/src/function/math/hypot.rs:
##########
@@ -0,0 +1,90 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, AsArray, Float64Array};
+use arrow::compute::kernels::arity::binary;
+use arrow::datatypes::{DataType, Float64Type};
+use datafusion_common::Result;
+use datafusion_common::utils::take_function_args;
+use datafusion_expr::{
+ ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+
+/// Spark-compatible `hypot` function.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#hypot>
+///
+/// Returns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow or
+/// underflow, matching Spark's use of `java.lang.Math.hypot`.
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkHypot {
+ signature: Signature,
+}
+
+impl Default for SparkHypot {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkHypot {
+ pub fn new() -> Self {
+ Self {
+ // Spark only defines hypot over doubles; `exact` makes coercion
+ // guarantee both inputs are Float64 before `invoke` runs.
+ signature: Signature::exact(
+ vec![DataType::Float64, DataType::Float64],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkHypot {
+ fn name(&self) -> &str {
+ "hypot"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
+ Ok(DataType::Float64)
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let num_rows = args.number_rows;
+ let [x, y] = take_function_args(self.name(), args.args)?;
+
+ // Broadcast scalars to arrays so one path covers every combination.
+ let x = x.to_array(num_rows)?;
Review Comment:
`make_scalar_function()` handles this for us
e.g.
```rust
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
make_scalar_function(hypot_array, vec![])(&args.args)
}
// outside impl
fn hypot_array(args: &[ArrayRef]) -> Result<ArrayRef> {
let [x, y] = take_function_args("hypot", args)?;
let x = x.as_primitive::<Float64Type>();
let y = y.as_primitive::<Float64Type>();
return Ok(Arc::new(binary(x, y, |a, b| a.hypot(b))))
}
```
##########
datafusion/spark/src/function/math/hypot.rs:
##########
@@ -0,0 +1,90 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, AsArray, Float64Array};
+use arrow::compute::kernels::arity::binary;
+use arrow::datatypes::{DataType, Float64Type};
+use datafusion_common::Result;
+use datafusion_common::utils::take_function_args;
+use datafusion_expr::{
+ ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+
+/// Spark-compatible `hypot` function.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#hypot>
+///
+/// Returns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow or
+/// underflow, matching Spark's use of `java.lang.Math.hypot`.
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkHypot {
+ signature: Signature,
+}
+
+impl Default for SparkHypot {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkHypot {
+ pub fn new() -> Self {
+ Self {
+ // Spark only defines hypot over doubles; `exact` makes coercion
+ // guarantee both inputs are Float64 before `invoke` runs.
Review Comment:
```suggestion
// Spark only defines hypot over doubles
```
extra detail unnecessary
##########
datafusion/sqllogictest/test_files/spark/math/hypot.slt:
##########
@@ -21,7 +21,45 @@
# For more information, please see:
# https://github.com/apache/datafusion/issues/15914
-## Original Query: SELECT hypot(3, 4);
-## PySpark 3.5.5 Result: {'HYPOT(3, 4)': 5.0, 'typeof(HYPOT(3, 4))': 'double', 'typeof(3)': 'int', 'typeof(4)': 'int'}
-#query
-#SELECT hypot(3::int, 4::int);
+# Scalar: classic Pythagorean triples (3-4-5, 5-12-13)
Review Comment:
could we add some more edge cases such as infinity inputs, nans, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org