You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/07/08 17:50:36 UTC

Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

viirya commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4917636970

   Reworked as suggested — the PR now adds a scalar-free `cdef _getitem_py(self, int64_t i)` on `Array` and the baseline `to_pylist` is a single loop over it. The base implementation is `GetScalar + Scalar.as_py`, so unspecialized types (dates/times/timestamps/decimals/dictionary/extension/...) behave exactly as before, and specializations exist for integers, floats, boolean, string/binary (+ large variants), list/large_list/fixed_size_list, map and struct. Nested types compose: a list row is built directly from the child's `_getitem_py` over its offset range, with the child wrapper cached on the parent array. `maps_as_pydicts != None` keeps the Scalar path since map→dict conversion has per-entry duplicate-key semantics.
   
   Your `int64` example: 4M `int64` with nulls goes from ~0.39 s to **0.028 s (~7 ns/element — on par with `ndarray.tolist`)**. Other numbers (M4 Max): flat string 4M 0.83→0.06 s; `list<string>` 2M 1.93→0.46 s; `list<list<int32>>` 1M 2.10→0.40 s; `struct<int64,string>` 1M 0.91→0.07 s; `map<string,int64>` 1M 2.77→0.74 s.
   
   Net diff is smaller than the previous approach (−104 lines) since the per-type `to_pylist` overrides are gone. Differential tests against the Scalar path (exact type equality, incl. slices, all-null, duplicate struct field names raising `ValueError`, strict-mode maps) pass, as does the pytest suite (1295 passed). PR description updated accordingly.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org