You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "vranes (via GitHub)" <gi...@apache.org> on 2026/10/13 19:21:51 UTC

[PR] [SPARK-60223][SQL] Make the ASOF join configs internal [spark]

vranes opened a new pull request, #59440:
URL: https://github.com/apache/spark/pull/59440

   ### What changes were proposed in this pull request?
   
   This PR marks two ASOF join configs as internal:
   
   - `spark.sql.join.asofJoin.enabled`
   - `spark.sql.join.sortMergeAsOfJoin.enabled`
   
   It also removes the references to `spark.sql.join.asofJoin.enabled` from the ASOF JOIN SQL reference page and from the SQL migration guide.
   
   ### Why are the changes needed?
   
   Both configs are feature gates that are off by default. Users are not expected to set them, so they must not appear in the generated SQL configuration docs or in the `SET -v` output.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. Both configs were added in 4.3.0, which is not released yet.
   
   ### How was this patch tested?
   
   Existing tests. The change only adds `.internal()` to the two config definitions and updates the docs.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60223][SQL] Make the ASOF join configs internal [spark]

Posted by "vranes (via GitHub)" <gi...@apache.org>.
vranes commented on code in PR #59440:
URL: https://github.com/apache/spark/pull/59440#discussion_r4268441801


##########
docs/sql-ref-syntax-qry-select-asof-join.md:
##########
@@ -28,10 +28,6 @@ right relation. The match is the closest row on the right that satisfies a requi
 `ASOF JOIN` is asymmetric: the left relation drives per-row lookups into the right,
 and reversing the operands changes the result.
 
-`ASOF JOIN` is disabled by default. Set `spark.sql.join.asofJoin.enabled` to `true`

Review Comment:
   I just thought we should remove this instruction while the feature is not completely ready yet, or even the whole doc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60223][SQL] Make the ASOF join configs internal [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #59440:
URL: https://github.com/apache/spark/pull/59440#discussion_r4265523009


##########
docs/sql-migration-guide.md:
##########
@@ -51,7 +51,6 @@ license: |
 ## Upgrading from Spark SQL 4.2 to 4.3
 
 - Since Spark 4.3, map-typed arguments to distinct aggregates are normalized by key order. As a result, `COUNT(DISTINCT m)` treats maps with the same entries in different orders as the same value, and aggregates that return distinct input values, such as `COLLECT_LIST(DISTINCT m)`, return maps sorted by key.
-- Since Spark 4.3, [ASOF JOIN](sql-ref-syntax-qry-select-asof-join.html) is available as an opt-in SQL feature gated by `spark.sql.join.asofJoin.enabled` (default `false`). When disabled, `ASOF JOIN` fails at parse time with `UNSUPPORTED_FEATURE.ASOF_JOIN`.

Review Comment:
   I agree with this removal. This is a new feature and it definitely won't break people during new spark version upgrade.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60223][SQL] Make the ASOF join configs internal [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #59440:
URL: https://github.com/apache/spark/pull/59440#discussion_r4265527307


##########
docs/sql-ref-syntax-qry-select-asof-join.md:
##########
@@ -28,10 +28,6 @@ right relation. The match is the closest row on the right that satisfies a requi
 `ASOF JOIN` is asymmetric: the left relation drives per-row lookups into the right,
 and reversing the operands changes the result.
 
-`ASOF JOIN` is disabled by default. Set `spark.sql.join.asofJoin.enabled` to `true`

Review Comment:
   Why we remove this? People following this doc won't  be able to run ASOF JOIN then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60223][SQL] Make the ASOF join configs internal [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #59440:
URL: https://github.com/apache/spark/pull/59440#issuecomment-6169788907

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/52d0a661d5244e0f8053e08e770a15f468f8eb1a
   - merged into branch-4.x https://github.com/apache/spark/commit/08270e3c200f98e2e496390793ad9b2e20a1398c
   - merged into branch-4.3 https://github.com/apache/spark/commit/5192e6dfb11c568c54811e032a004c8f890a0f84
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60223][SQL] Make the ASOF join configs internal [spark]

Posted by "vranes (via GitHub)" <gi...@apache.org>.
vranes commented on code in PR #59440:
URL: https://github.com/apache/spark/pull/59440#discussion_r4268441801


##########
docs/sql-ref-syntax-qry-select-asof-join.md:
##########
@@ -28,10 +28,6 @@ right relation. The match is the closest row on the right that satisfies a requi
 `ASOF JOIN` is asymmetric: the left relation drives per-row lookups into the right,
 and reversing the operands changes the result.
 
-`ASOF JOIN` is disabled by default. Set `spark.sql.join.asofJoin.enabled` to `true`

Review Comment:
   I just thought we should remove this instruction while the feature is not completely ready yet, or even the whole doc



##########
docs/sql-ref-syntax-qry-select-asof-join.md:
##########
@@ -28,10 +28,6 @@ right relation. The match is the closest row on the right that satisfies a requi
 `ASOF JOIN` is asymmetric: the left relation drives per-row lookups into the right,
 and reversing the operands changes the result.
 
-`ASOF JOIN` is disabled by default. Set `spark.sql.join.asofJoin.enabled` to `true`

Review Comment:
   Reverted this change, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60223][SQL] Make the ASOF join configs internal [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #59440: [SPARK-60223][SQL] Make the ASOF join configs internal
URL: https://github.com/apache/spark/pull/59440


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org