You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "brijrajk (via GitHub)" <gi...@apache.org> on 2026/06/01 19:23:00 UTC
[PR] [SPARK-56561][PYTHON][DOCS] Document order preservation for array_distinct, array_intersect, array_union, array_except [spark]
brijrajk opened a new pull request, #56250:
URL: https://github.com/apache/spark/pull/56250
### What changes were proposed in this pull request?
Added explicit order-preservation notes to the docstrings of four array set
functions in PySpark: `array_distinct`, `array_intersect`, `array_union`, and
`array_except`. Previously the docstrings were silent on whether input order is
preserved, leaving users to guess or test empirically.
The additions are:
- `array_distinct`: *preserving the order of first occurrence*
- `array_intersect`: *preserving the order of elements from col1*
- `array_union`: *preserving the order from col1 followed by elements in col2 not in col1*
- `array_except`: *preserving the order of elements from col1*
### Why are the changes needed?
Order-preservation is a useful guarantee for users composing these functions in
pipelines — e.g. knowing that `array_distinct` keeps the first occurrence means
there is no need to sort afterwards to get a stable result. This is useful to
cite in code reviews and helps inform AI coding assistants as noted in
SPARK-56561.
Fixes https://issues.apache.org/jira/browse/SPARK-56561
### Does this PR introduce _any_ user-facing change?
No. Documentation (docstring) only — no logic changes.
### How was this patch tested?
No logic change. Confirmed order-preservation behavior by reading the
`ArraySetLike` trait and `ArrayDistinct`, `ArrayIntersect`, `ArrayUnion`, and
`ArrayExcept` implementations in `collectionOperations.scala`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude (Anthropic)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56561][PYTHON][DOCS] Document order preservation for array_distinct, array_intersect, array_union, array_except [spark]
Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk commented on PR #56250:
URL: https://github.com/apache/spark/pull/56250#issuecomment-4671159887
Thanks for pointing that out, @shrirangmhalgi. I reviewed PR #55549 and the feedback from @sarutak there directly applies here as well — documenting order preservation as an API guarantee would constrain future optimizations (e.g. sort-based deduplication, parallelization), and the original PRs for these functions (SPARK-23913/23914/23915) explicitly stated that element order in the result is not defined. Closing this PR accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56561][PYTHON][DOCS] Document order preservation for array_distinct, array_intersect, array_union, array_except [spark]
Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk closed pull request #56250: [SPARK-56561][PYTHON][DOCS] Document order preservation for array_distinct, array_intersect, array_union, array_except
URL: https://github.com/apache/spark/pull/56250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56561][PYTHON][DOCS] Document order preservation for array_distinct, array_intersect, array_union, array_except [spark]
Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on PR #56250:
URL: https://github.com/apache/spark/pull/56250#issuecomment-4667991103
FYI - there was a prior attempt at this in #55549. Might be worth checking if the same concerns apply here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org