[I] [Python] Support array indices in pc.list_element [arrow]

["Aokizy2 (via GitHub)" <gi...@apache.org>]

Aokizy2 opened a new issue, #48349:
URL: https://github.com/apache/arrow/issues/48349

   ### Describe the enhancement requested
   
   # Description:
   pc.list_element(array, index) only accepts scalar integer indices. This prevents efficient vectorized operations. For example, to get the last element from each list:
   
   ```python
   import pyarrow as pa
   import pyarrow.compute as pc
   arr = pa.array([[1, 2, 3], [4, 5], [6, 7, 8, 9]])
   ```
   # Desired but not supported:
   ```python
   indices = pa.array([2, 1, 3])  # or [-1, -1, -1]
   result = pc.list_element(arr, indices)  # Should return [3, 5, 9]
   ```
   # Use Case:
   pandas needs this for efficient .list[-1] operations with pyarrow-backed arrays. Currently, pandas must use slow Python loops as a workaround.
   
   # Proposed Solution:
   Extend pc.list_element to accept array indices (like other compute functions do). This would enable vectorized element access from lists.
   
   # Related Issues:
   
   pandas issue: (#63221)
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Support array indices in pc.list_element [arrow]

["Aokizy2 (via GitHub)" <gi...@apache.org>]

Aokizy2 commented on issue #48349:
URL: https://github.com/apache/arrow/issues/48349#issuecomment-3617298555

   Thank you for the technical solution! This is indeed a clever workaround.
   
   However, for pandas production use, I'm concerned about:
   1. **Performance**: Group-by operations may not be efficient enough for large datasets
   2. **Simplicity**: `.list[-1]` should be a simple operation from the user's perspective  
   3. **Maintainability**: Complex implementations increase long-term maintenance costs
   
   I think it would be better to have native support for array indices in pyarrow, which would provide better performance and cleaner implementation. @dangotbanned 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Support array indices in pc.list_element [arrow]

["rhshadrach (via GitHub)" <gi...@apache.org>]

rhshadrach commented on issue #48349:
URL: https://github.com/apache/arrow/issues/48349#issuecomment-3620357000

   This is a duplicate of https://github.com/apache/arrow/issues/40033


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Support array indices in pc.list_element [arrow]

["dangotbanned (via GitHub)" <gi...@apache.org>]

dangotbanned commented on issue #48349:
URL: https://github.com/apache/arrow/issues/48349#issuecomment-3617144584

   > pandas needs this for efficient .list[-1] operations with pyarrow-backed arrays. Currently, pandas must use slow Python loops as a workaround.
   
   For this specific case, you can do:
   
   ```py
   import pyarrow as pa
   import pyarrow.compute as pc
   
   arr = pa.array([[1, 2, 3], [4, 5], [6, 7, 8, 9]])
   
   parent_indices = arr.value_parent_indices()
   
   # NOTE: or `np.arange` if `pyarrow<21`
   value_indices = pa.arange(0, len(parent_indices), 1)
   
   # NOTE: These are the defaults, but you can preserve nulls by changing this
   options = pc.ScalarAggregateOptions(skip_nulls=True, min_count=1)
   
   grouped_last = (
       pa.Table.from_arrays([parent_indices, value_indices], ["parent", "value"])
       .group_by("parent", use_threads=False)
       .aggregate([("value", "hash_last", options)])
   )
   arr.values.take(grouped_last.column(1)).to_pylist()
   ```
   
   ```py
   [3, 5, 9]
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org