You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/24 22:59:38 UTC
[PR] feat: native pow implementation compatible with Spark [datafusion-comet]
andygrove opened a new pull request, #5033:
URL: https://github.com/apache/datafusion-comet/pull/5033
## Which issue does this PR close?
There is no dedicated tracking issue. The native path's incompatibility stems from the upstream DataFusion behavior in apache/datafusion#22598.
Closes #.
## Rationale for this change
Spark's `Pow` expression delegates to Java's `Math.pow`, which returns `Infinity` for a zero base raised to a negative exponent (`pow(0, -1)`). Comet's native path routed `pow` to DataFusion's `PowerFunc`, whose `float64_power_checked` instead **errors** on that input (apache/datafusion#22598). Because of that single edge case, `CometPow` was marked `Unsupported`, so any query using `pow`/`power` fell back to Spark for that part of the plan.
Rust's `f64::powf` follows the same IEEE-754 pow semantics as Java's `Math.pow` (including `0^-1 = Infinity`, `(-0)^-1 = -Infinity`, and the `NaN`/`Infinity` cases), so a thin native kernel over `powf` matches Spark exactly and lets `pow` run natively by default instead of falling back.
## What changes are included in this PR?
- Add a `spark_pow` native kernel (`native/spark-expr/src/math_funcs/pow.rs`) that computes `base.powf(exp)` element-wise with null propagation, covering all Array/Scalar argument combinations.
- Register `"pow"` to `spark_pow` in `create_comet_physical_fun_with_eval_mode`, so it no longer falls through to DataFusion's checked `power`, plus the module/re-export wiring.
- Simplify `CometPow` to a plain `Compatible` serde (drop the `Unsupported`/"Power has correctness issues" reporting).
## How are these changes tested?
- Rust unit tests in `pow.rs` covering the basic cases, `pow(0, -1) == Infinity`, null propagation, and the scalar-base path.
- The `expressions/math/pow.sql` SQL file test (run via `CometSqlFileTestSuite`) now asserts native execution matches Spark across zero-base-negative-exponent, `NaN`, `Infinity`, null, and column/literal argument combinations, replacing the previous `expect_fallback` assertions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make native pow implementation compatible with Spark [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #5033:
URL: https://github.com/apache/datafusion-comet/pull/5033#discussion_r3648795593
##########
spark/src/test/resources/sql-tests/expressions/math/pow.sql:
##########
@@ -15,27 +15,25 @@
-- specific language governing permissions and limitations
-- under the License.
+-- pow runs natively and matches Spark exactly, including edge cases such as pow(0, -1) = Infinity.
+
statement
CREATE TABLE test_pow(base double, exp double) USING parquet
statement
INSERT INTO test_pow VALUES (0.0, -1), (2.0, 3.0), (0.0, 0.0), (-1.0, 2.0), (-1.0, 0.5), (2.0, -1.0), (NULL, 2.0), (2.0, NULL), (cast('NaN' as double), 2.0), (cast('Infinity' as double), 2.0), (2.0, cast('Infinity' as double))
Review Comment:
negative infinity can be interesting too. we probably need a bit more combinations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make native pow implementation compatible with Spark [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5033:
URL: https://github.com/apache/datafusion-comet/pull/5033#discussion_r3650404916
##########
spark/src/test/resources/sql-tests/expressions/math/pow.sql:
##########
@@ -15,27 +15,25 @@
-- specific language governing permissions and limitations
-- under the License.
+-- pow runs natively and matches Spark exactly, including edge cases such as pow(0, -1) = Infinity.
+
statement
CREATE TABLE test_pow(base double, exp double) USING parquet
statement
INSERT INTO test_pow VALUES (0.0, -1), (2.0, 3.0), (0.0, 0.0), (-1.0, 2.0), (-1.0, 0.5), (2.0, -1.0), (NULL, 2.0), (2.0, NULL), (cast('NaN' as double), 2.0), (cast('Infinity' as double), 2.0), (2.0, cast('Infinity' as double))
Review Comment:
Thanks for the review @kazuyukitanimura. I added more tests as suggested and it did expose a bug, which is now fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make native pow implementation compatible with Spark [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5033:
URL: https://github.com/apache/datafusion-comet/pull/5033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make native pow implementation compatible with Spark [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #5033:
URL: https://github.com/apache/datafusion-comet/pull/5033#discussion_r3648789744
##########
spark/src/test/resources/sql-tests/expressions/math/pow.sql:
##########
@@ -15,27 +15,25 @@
-- specific language governing permissions and limitations
-- under the License.
+-- pow runs natively and matches Spark exactly, including edge cases such as pow(0, -1) = Infinity.
+
statement
CREATE TABLE test_pow(base double, exp double) USING parquet
statement
INSERT INTO test_pow VALUES (0.0, -1), (2.0, 3.0), (0.0, 0.0), (-1.0, 2.0), (-1.0, 0.5), (2.0, -1.0), (NULL, 2.0), (2.0, NULL), (cast('NaN' as double), 2.0), (cast('Infinity' as double), 2.0), (2.0, cast('Infinity' as double))
Review Comment:
What about negative zero cases? and subnormal numbers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org