You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cloud-fan (via GitHub)" <gi...@apache.org> on 2026/09/28 05:18:47 UTC
[PR] [SPARK-59673][SQL][FOLLOWUP] Restore zero as NAAJ broadcast threshold disable sentinel [spark]
cloud-fan opened a new pull request, #59091:
URL: https://github.com/apache/spark/pull/59091
### What changes were proposed in this pull request?
This follow-up restores the three-way contract for `spark.sql.optimizer.nullAwareAntiJoinBroadcastThreshold`:
- A negative value allows the specialized null-aware anti join broadcast hash path without a size limit.
- Zero always disables that hash path.
- A positive value admits the path when either the dedicated threshold or the applicable automatic broadcast threshold allows it.
It updates the shared join-selection logic, configuration documentation, and optimizer and execution tests.
### Why are the changes needed?
SPARK-36082 established zero as the sentinel for disabling broadcast NAAJ. SPARK-59673 made the automatic broadcast threshold a floor, but unintentionally allowed that threshold to override zero. This restores the existing disable sentinel while preserving the automatic-threshold floor for positive dedicated thresholds.
### Does this PR introduce _any_ user-facing change?
Yes. On current master, setting `spark.sql.optimizer.nullAwareAntiJoinBroadcastThreshold` to zero once again disables the specialized NAAJ broadcast hash path even when an automatic broadcast threshold is positive. This corrects behavior introduced by the unreleased SPARK-59673 change.
### How was this patch tested?
Updated `JoinSelectionHelperSuite`, `LeftSemiAntiJoinPushDownSuite`, and `JoinSuite` to cover the zero disable sentinel and the positive-threshold behavior.
The following focused suites were attempted:
- `catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.JoinSelectionHelperSuite`
- `catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.LeftSemiAntiJoinPushDownSuite`
- `sql/testOnly org.apache.spark.sql.JoinSuite -- -z SPARK-36082`
The build could not reach compilation because the environment could not resolve `repo.maven.apache.org` while fetching Maven dependencies. `git diff --check`, changed-file line-length checks, and changed-file non-ASCII checks passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59673][SQL][FOLLOWUP] Restore zero as NAAJ broadcast threshold disable sentinel [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #59091: [SPARK-59673][SQL][FOLLOWUP] Restore zero as NAAJ broadcast threshold disable sentinel
URL: https://github.com/apache/spark/pull/59091
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59673][SQL][FOLLOWUP] Restore zero as NAAJ broadcast threshold disable sentinel [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #59091:
URL: https://github.com/apache/spark/pull/59091#issuecomment-5864022559
wrong conclusion, closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org