You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "brijrajk (via GitHub)" <gi...@apache.org> on 2026/06/01 19:23:00 UTC

[PR] [SPARK-56561][PYTHON][DOCS] Document order preservation for array_distinct, array_intersect, array_union, array_except [spark]

brijrajk opened a new pull request, #56250:
URL: https://github.com/apache/spark/pull/56250

   ### What changes were proposed in this pull request?
   
   Added explicit order-preservation notes to the docstrings of four array set
   functions in PySpark: `array_distinct`, `array_intersect`, `array_union`, and
   `array_except`. Previously the docstrings were silent on whether input order is
   preserved, leaving users to guess or test empirically.
   
   The additions are:
   
   - `array_distinct`: *preserving the order of first occurrence*
   - `array_intersect`: *preserving the order of elements from col1*
   - `array_union`: *preserving the order from col1 followed by elements in col2 not in col1*
   - `array_except`: *preserving the order of elements from col1*
   
   ### Why are the changes needed?
   
   Order-preservation is a useful guarantee for users composing these functions in
   pipelines — e.g. knowing that `array_distinct` keeps the first occurrence means
   there is no need to sort afterwards to get a stable result. This is useful to
   cite in code reviews and helps inform AI coding assistants as noted in
   SPARK-56561.
   
   Fixes https://issues.apache.org/jira/browse/SPARK-56561
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. Documentation (docstring) only — no logic changes.
   
   ### How was this patch tested?
   
   No logic change. Confirmed order-preservation behavior by reading the
   `ArraySetLike` trait and `ArrayDistinct`, `ArrayIntersect`, `ArrayUnion`, and
   `ArrayExcept` implementations in `collectionOperations.scala`.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude (Anthropic)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56561][PYTHON][DOCS] Document order preservation for array_distinct, array_intersect, array_union, array_except [spark]

Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk commented on PR #56250:
URL: https://github.com/apache/spark/pull/56250#issuecomment-4671159887

   Thanks for pointing that out, @shrirangmhalgi. I reviewed PR #55549 and the feedback from @sarutak there directly applies here as well — documenting order preservation as an API guarantee would constrain future optimizations (e.g. sort-based deduplication, parallelization), and the original PRs for these functions (SPARK-23913/23914/23915) explicitly stated that element order in the result is not defined. Closing this PR accordingly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56561][PYTHON][DOCS] Document order preservation for array_distinct, array_intersect, array_union, array_except [spark]

Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk closed pull request #56250: [SPARK-56561][PYTHON][DOCS] Document order preservation for array_distinct, array_intersect, array_union, array_except
URL: https://github.com/apache/spark/pull/56250


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56561][PYTHON][DOCS] Document order preservation for array_distinct, array_intersect, array_union, array_except [spark]

Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on PR #56250:
URL: https://github.com/apache/spark/pull/56250#issuecomment-4667991103

   FYI - there was a prior attempt at this in #55549. Might be worth checking if the same concerns apply here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org