You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/06/30 18:13:33 UTC
[PR] feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates [datafusion-comet]
andygrove opened a new pull request, #4775:
URL: https://github.com/apache/datafusion-comet/pull/4775
## Which issue does this PR close?
Closes #4552.
## Rationale for this change
Comet already accelerates `regr_avgx`, `regr_avgy`, and `regr_count` (Spark rewrites those to `Average`/`Count`), but the remaining six SQL-standard linear regression aggregates fell back to Spark. This PR implements native support for `regr_slope`, `regr_intercept`, `regr_r2`, `regr_sxx`, `regr_syy`, and `regr_sxy`, so a query using any of them can run fully on Comet instead of falling back.
## What changes are included in this PR?
- A new native `Regr` aggregate UDF in `native/spark-expr/src/agg_funcs/regr.rs`. Rather than re-implementing the statistics, each function is composed from Comet's existing Spark-compatible `CovarianceAccumulator` and `VarianceAccumulator`. This keeps the partial aggregation state byte-compatible with the buffer layout Spark's planner declares for the partial to final shuffle:
- `regr_sxx` / `regr_syy` reach Comet as `RegrReplacement` (a `CentralMomentAgg`, 3-field buffer) and reuse the variance accumulator, evaluating to `m2`.
- `regr_sxy` (a population `Covariance`, 4-field buffer) reuses the covariance accumulator, evaluating to the co-moment `ck`.
- `regr_r2` (a `PearsonCorrelation`, 6-field buffer) composes covariance + two variances.
- `regr_slope` / `regr_intercept` (a declarative composite, 7-field buffer) compose population covariance + variance.
- `regr_r2` matches Spark's behavior of returning `1.0` when the dependent variable is constant but the independent variable varies (a perfect horizontal fit). This is the one case where Spark and DataFusion's `regr_r2` diverge (DataFusion returns `null`), so a Comet-specific accumulator is warranted.
- A new `Regr` protobuf message (with a `RegrType` enum) wired through `QueryPlanSerde` and the native `PhysicalPlanner`.
- Scala serde for `RegrSlope`, `RegrIntercept`, `RegrR2`, `RegrSXY`, and `RegrReplacement` (the rewrite target for `regr_sxx`/`regr_syy`).
- Updated the expression support status in the user guide.
This work was scaffolded using the `implement-comet-expression` project skill.
## How are these changes tested?
- Expanded the existing `spark/src/test/resources/sql-tests/expressions/aggregate/regr.sql` so all six functions are verified to run natively and match Spark (previously `spark_answer_only`). New cases cover NULL pairs, single-pair input, constant independent variable (slope/intercept/r2 to NULL), constant dependent variable (r2 to 1.0), grouped aggregation, and literal/column argument mixes.
- Added Rust unit tests in `regr.rs` covering perfect-fit slope/intercept/r2, the constant-y and constant-x edge cases, single-pair and empty input, NULL-pair skipping, the raw moments, and partial-state merge across batches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates [datafusion-comet]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #4775:
URL: https://github.com/apache/datafusion-comet/pull/4775#discussion_r3900863955
##########
native/proto/src/proto/expr.proto:
##########
@@ -263,6 +264,36 @@ message Correlation {
DataType datatype = 4;
}
+// Simple linear regression aggregates (regr_slope, regr_intercept, regr_r2,
+// regr_sxx, regr_syy, regr_sxy). child1 is the dependent variable (y) and
+// child2 is the independent variable (x).
+message Regr {
+ enum RegrType {
+ SLOPE = 0;
+ INTERCEPT = 1;
+ R2 = 2;
+ SXX = 3;
+ SYY = 4;
+ SXY = 5;
+ }
+ Expr child1 = 1;
+ Expr child2 = 2;
+ RegrType regr_type = 3;
+ DataType datatype = 4;
+ // Only consulted for SLOPE and INTERCEPT. When true (Spark 3.5+), VariancePop(x)
+ // is computed only over rows where both y and x are non-null. When false
+ // (Spark 3.4), VariancePop(x) includes every row where x is non-null even if y
+ // is null, matching the pre-fix Spark 3.4 semantics.
+ bool filter_var_by_pair_nulls = 5;
+ // Only consulted for R2. Spark swapped the degenerate-case handling of
+ // regr_r2. When true, a constant dependent variable (m2(y) = 0) returns 1.0
+ // and a constant independent variable (m2(x) = 0) returns null. When false
+ // those two cases are reversed. The swap is present in the Spark versions
+ // Comet builds against for 3.5 and later (3.5.9, 4.0.3+, 4.1, 4.2) but not
Review Comment:
The list can become style. This comment can be simplified to
```
// The swap is present in the Spark versions Comet builds against for 3.5+, but not in 3.4.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #4775:
URL: https://github.com/apache/datafusion-comet/pull/4775#issuecomment-5480805607
> @andygrove Can you rebase this PR?
@manuzhang Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates [datafusion-comet]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #4775:
URL: https://github.com/apache/datafusion-comet/pull/4775#discussion_r3900887644
##########
spark/src/main/scala/org/apache/comet/serde/aggregates.scala:
##########
@@ -763,6 +763,131 @@ object CometCorr extends CometAggregateExpressionSerde[Corr] {
}
}
+/**
+ * Shared serialization for the simple linear regression aggregates. `child1` is the dependent
+ * variable (y) and `child2` is the independent variable (x), matching the native accumulator's
+ * `regr_*(y, x)` convention.
+ */
+trait CometRegrBase {
+ def convertRegr(
+ aggExpr: AggregateExpression,
+ regrType: ExprOuterClass.Regr.RegrType,
+ y: Expression,
+ x: Expression,
+ inputs: Seq[Attribute],
+ binding: Boolean): Option[ExprOuterClass.AggExpr] = {
+ val child1Expr = exprToProto(y, inputs, binding)
+ val child2Expr = exprToProto(x, inputs, binding)
+ val dataType = serializeDataType(DoubleType)
+
+ if (child1Expr.isDefined && child2Expr.isDefined && dataType.isDefined) {
+ val builder = ExprOuterClass.Regr.newBuilder()
+ builder.setChild1(child1Expr.get)
+ builder.setChild2(child2Expr.get)
+ builder.setRegrType(regrType)
+ builder.setDatatype(dataType.get)
+ // Spark 3.5 fixed regr_slope/regr_intercept so VariancePop(x) only counts
Review Comment:
Maybe better (describing what not how)
```
VariancePop(x) only counts rows where both y and x are non-null in Spark 3.5+, but counts every row where x is non-null in 3.4
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #4775:
URL: https://github.com/apache/datafusion-comet/pull/4775#issuecomment-5539461962
Merged. Thanks @manuzhang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #4775:
URL: https://github.com/apache/datafusion-comet/pull/4775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates [datafusion-comet]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #4775:
URL: https://github.com/apache/datafusion-comet/pull/4775#issuecomment-5480330693
@andygrove Can you rebase this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates [datafusion-comet]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #4775:
URL: https://github.com/apache/datafusion-comet/pull/4775#discussion_r3900869807
##########
spark/src/main/scala/org/apache/comet/serde/aggregates.scala:
##########
@@ -763,6 +763,131 @@ object CometCorr extends CometAggregateExpressionSerde[Corr] {
}
}
+/**
+ * Shared serialization for the simple linear regression aggregates. `child1` is the dependent
+ * variable (y) and `child2` is the independent variable (x), matching the native accumulator's
+ * `regr_*(y, x)` convention.
+ */
+trait CometRegrBase {
+ def convertRegr(
+ aggExpr: AggregateExpression,
+ regrType: ExprOuterClass.Regr.RegrType,
+ y: Expression,
+ x: Expression,
+ inputs: Seq[Attribute],
+ binding: Boolean): Option[ExprOuterClass.AggExpr] = {
+ val child1Expr = exprToProto(y, inputs, binding)
+ val child2Expr = exprToProto(x, inputs, binding)
+ val dataType = serializeDataType(DoubleType)
+
+ if (child1Expr.isDefined && child2Expr.isDefined && dataType.isDefined) {
+ val builder = ExprOuterClass.Regr.newBuilder()
+ builder.setChild1(child1Expr.get)
+ builder.setChild2(child2Expr.get)
+ builder.setRegrType(regrType)
+ builder.setDatatype(dataType.get)
+ // Spark 3.5 fixed regr_slope/regr_intercept so VariancePop(x) only counts
+ // rows where both y and x are non-null. Spark 3.4 counts every row where x
+ // is non-null. The native accumulator only consults this for slope/intercept.
+ builder.setFilterVarByPairNulls(isSpark35Plus)
+ // Spark swapped regr_r2's degenerate-case handling: a constant dependent
+ // variable now yields 1.0 (was null) and a constant independent variable
+ // yields null (was 1.0). The swap is present in the Spark versions Comet
+ // builds against for 3.5 and later (3.5.9, 4.0.3+, 4.1, 4.2) but not in 3.4.
Review Comment:
same here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org