You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/07/01 23:30:34 UTC

[PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

viirya opened a new pull request, #50327:
URL: https://github.com/apache/arrow/pull/50327

   ### Rationale for this change
   
   `Array.to_pylist()` on list-typed arrays is 2.5–10x slower than converting the same array via `to_pandas()` and rebuilding Python lists from the resulting numpy arrays, even though `to_pylist` does strictly less work. The cause is the per-element conversion loop (`[x.as_py() for x in self]`): every row allocates a C++ Scalar (`Array::GetScalar`), a Python Scalar wrapper and, for list types, a Python Array wrapper for the row's values slice plus a fresh generator before recursing per element. Besides the allocation cost, these GC-tracked wrappers repeatedly trigger CPython collections that traverse the ever-growing result list (~20% of runtime in a `sample` profile; details in #50326).
   
   This hit Apache Spark when it enabled Arrow-serialized Python UDFs by default (apache/spark#56940, apache/spark#56943); working around it via `to_pandas()` was rejected there because the pandas detour coerces `list<int32>` with nulls to numpy `float64` (`[1., nan, 3.]` instead of `[1, None, 3]`).
   
   Benchmarks (macOS arm64, Python 3.11; 2M rows of 2-element lists / 1M rows of nested lists):
   
   | benchmark | before | after | speedup |
   |---|---|---|---|
   | `list<string>` to_pylist | 1.93 s | 0.34 s | 5.7x |
   | `list<list<int32>>` to_pylist | 2.10 s | 0.65 s | 3.2x |
   | flat `string` to_pylist (4M) | 0.83 s | 0.05 s | 16x |
   
   For reference, the pandas detour (`to_pandas()` + per-row `tolist()`) takes 0.75 s on the `list<string>` case, so `to_pylist` goes from 2.5x slower to ~2.2x faster.
   
   ### What changes are included in this PR?
   
   Bulk `to_pylist` overrides in `array.pxi`:
   
   - `ListArray` / `LargeListArray` / `FixedSizeListArray`: convert the referenced range of child values with a single recursive `to_pylist` call, then slice the resulting Python list per row using the raw offsets and the validity bitmap. No per-row Scalar, Python Array wrapper or generator. `MapArray` explicitly keeps the generic scalar-based path (association-tuple / `maps_as_pydicts` duplicate-key semantics), as do the list-view types (overlapping views must not share sublist objects).
   - `StringArray` / `LargeStringArray`: decode values directly from the data buffer (`GetValue` + `PyUnicode_DecodeUTF8`), which matches `StringScalar.as_py` (= `str(buf, 'utf8')`) exactly.
   
   Output is unchanged, including exact element types: `None` stays `None`, values inside numeric lists stay Python ints (never floats/NaN), strings/bytes are unchanged. `ChunkedArray.to_pylist`, `Table.to_pylist` and `ListScalar.as_py` delegate to `Array.to_pylist` and pick up the speedup automatically.
   
   Follow-up candidates (not in this PR): leaf fast paths for primitive/binary/view types, a bulk path for maps and structs, or a general C++ `ToPyList` visitor covering all types.
   
   ### Are these changes tested?
   
   - New `test_to_pylist_bulk_paths` compares the bulk paths against the per-scalar conversion (`[x.as_py() for x in arr]`) for list/large_list/fixed_size_list/nested/map/string/large_string arrays, including sliced, empty and all-null arrays, and asserts exact element types for `list<int32>` with nulls.
   - Existing suites pass: `test_array.py`, `test_scalars.py`, `test_convert_builtin.py`, `test_table.py` (1209 passed locally).
   - Additionally verified with a randomized differential test (8 leaf types x list/large_list/fixed_size_list/map, nested lists, list\<struct\>, list\<map\>, slices, both `maps_as_pydicts` modes, multibyte strings) with exact-type comparison: no differences.
   
   ### Are there any user-facing changes?
   
   No behavior changes, only performance: `to_pylist()` on list-like and string arrays is several times faster.
   
   * GitHub Issue: #50326
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3552797257


##########
python/pyarrow/array.pxi:
##########
@@ -2458,6 +2476,34 @@ cdef class NumericArray(Array):
     A base class for Arrow numeric arrays.
     """
 
+    cdef object _getitem_py(self, int64_t i):
+        cdef Type tid = self.ap.type_id()
+        if self.ap.IsNull(i):
+            return None
+        if tid == _Type_INT64:
+            return (<CInt64Array*> self.ap).Value(i)
+        elif tid == _Type_INT32:
+            return (<CInt32Array*> self.ap).Value(i)

Review Comment:
   Reordered to the regular `int8..uint64, float, double` sequence in 3d303ce487.



##########
python/pyarrow/array.pxi:
##########
@@ -4006,6 +4105,15 @@ cdef class LargeStringArray(Array):
     Concrete class for Arrow arrays of large string (or utf8) data type.
     """
 
+    cdef object _getitem_py(self, int64_t i):
+        cdef:
+            int64_t length
+            const uint8_t* data
+        if self.ap.IsNull(i):
+            return None
+        data = (<CLargeStringArray*> self.ap).GetValue(i, &length)
+        return cp.PyUnicode_DecodeUTF8(<const char*> data, length, NULL)

Review Comment:
   Thanks for the correction — I stand corrected on 32-bit support; agreed it's moot either way since such a value can't exist in a 32-bit address space.



##########
python/pyarrow/array.pxi:
##########
@@ -4229,6 +4357,30 @@ cdef class StructArray(Array):
     Concrete class for Arrow arrays of a struct data type.
     """
 
+    cdef object _getitem_py(self, int64_t i):
+        if self.ap.IsNull(i):
+            return None
+        cdef int64_t k, num_fields = self.type.num_fields
+        if self._children_cache is None:
+            names = [self.type.field(k).name for k in range(num_fields)]
+            if len(set(names)) != len(names):
+                # StructScalar.as_py raises ValueError on duplicate field
+                # names; mark the cache so we take the Scalar path below.
+                self._children_cache = (None, None)

Review Comment:
   Done — it now raises the same `ValueError` as `StructScalar.as_py` directly (the cache/fallback dance is gone), and the test asserts the message.



##########
python/pyarrow/array.pxi:
##########
@@ -1864,7 +1864,19 @@ cdef class Array(_PandasConvertible):
         lst : list
         """
         self._assert_cpu()
-        return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        cdef int64_t i, n = self.length()
+        if maps_as_pydicts is not None:
+            # Converting maps to dicts has per-entry semantics (duplicate-key
+            # detection); use the Scalar-based conversion for exact behavior.
+            return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        return [self._getitem_py(i) for i in range(n)]
+
+    cdef object _getitem_py(self, int64_t i):

Review Comment:
   Filed GH-50448 for the per-range conversion (folding in the dispatch-hoisting and null-check ideas from this thread as well), and added a TODO pointing at it.



##########
python/pyarrow/array.pxi:
##########
@@ -4006,6 +4105,15 @@ cdef class LargeStringArray(Array):
     Concrete class for Arrow arrays of large string (or utf8) data type.
     """
 
+    cdef object _getitem_py(self, int64_t i):
+        cdef:
+            int64_t length
+            const uint8_t* data
+        if self.ap.IsNull(i):
+            return None
+        data = (<CLargeStringArray*> self.ap).GetValue(i, &length)

Review Comment:
   Nice — done. Switched string/binary (+ large variants) to `GetView(i)`; it also made `StringViewArray`/`BinaryViewArray` specializations trivial, so those are included now too.



##########
python/pyarrow/tests/test_array.py:
##########
@@ -465,6 +465,56 @@ def test_array_getitem_numpy_scalars():
         assert arr[np.int32(idx)].as_py() == lst[idx]
 
 
+def test_to_pylist_bulk_paths():
+    # GH-50326: to_pylist converts through scalar-free _getitem_py
+    # specializations; the result must match the per-scalar conversion
+    # exactly.
+    arrays = [
+        pa.array([[1, None, 3], None, [], [4]], type=pa.list_(pa.int32())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.list_(pa.string())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.large_list(pa.large_string())),
+        pa.array([[1, None], None, [3, 4]], type=pa.list_(pa.int32(), 2)),
+        pa.array([[[1], [2, None]], None, [None, [3]]],
+                 type=pa.list_(pa.list_(pa.int32()))),
+        pa.array([[("k1", 1), ("k2", None)], None, []],
+                 type=pa.map_(pa.string(), pa.int32())),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.string()),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.large_string()),
+        pa.array([b"a\x00b", None, b"", b"\xff"], type=pa.binary()),
+        pa.array([b"a\x00b", None, b""], type=pa.large_binary()),
+        pa.array([[b"x", None, b"\x00y"], None, []],
+                 type=pa.list_(pa.binary())),
+        pa.array([1, None, -(2**62), 2**62], type=pa.int64()),
+        pa.array([0, None, 2**63 + 7], type=pa.uint64()),
+        pa.array([-128, 127, None], type=pa.int8()),
+        pa.array([1.5, None, -0.5], type=pa.float64()),
+        pa.array([1.5, None], type=pa.float32()),
+        pa.array([True, None, False], type=pa.bool_()),
+        pa.array([{"a": 1, "b": "x"}, None, {"a": None, "b": None}],
+                 type=pa.struct([("a", pa.int32()), ("b", pa.string())])),
+        pa.array([], type=pa.list_(pa.int32())),
+        pa.array([None, None], type=pa.list_(pa.string())),
+    ]

Review Comment:
   Added `binary_view` and `string_view` cases — and since `GetView` covers the view types, they now take real fast paths instead of the Scalar fallback.



##########
python/pyarrow/tests/test_array.py:
##########
@@ -465,6 +465,56 @@ def test_array_getitem_numpy_scalars():
         assert arr[np.int32(idx)].as_py() == lst[idx]
 
 
+def test_to_pylist_bulk_paths():
+    # GH-50326: to_pylist converts through scalar-free _getitem_py
+    # specializations; the result must match the per-scalar conversion
+    # exactly.
+    arrays = [
+        pa.array([[1, None, 3], None, [], [4]], type=pa.list_(pa.int32())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.list_(pa.string())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.large_list(pa.large_string())),
+        pa.array([[1, None], None, [3, 4]], type=pa.list_(pa.int32(), 2)),
+        pa.array([[[1], [2, None]], None, [None, [3]]],
+                 type=pa.list_(pa.list_(pa.int32()))),
+        pa.array([[("k1", 1), ("k2", None)], None, []],
+                 type=pa.map_(pa.string(), pa.int32())),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.string()),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.large_string()),
+        pa.array([b"a\x00b", None, b"", b"\xff"], type=pa.binary()),
+        pa.array([b"a\x00b", None, b""], type=pa.large_binary()),
+        pa.array([[b"x", None, b"\x00y"], None, []],
+                 type=pa.list_(pa.binary())),
+        pa.array([1, None, -(2**62), 2**62], type=pa.int64()),
+        pa.array([0, None, 2**63 + 7], type=pa.uint64()),
+        pa.array([-128, 127, None], type=pa.int8()),
+        pa.array([1.5, None, -0.5], type=pa.float64()),
+        pa.array([1.5, None], type=pa.float32()),
+        pa.array([True, None, False], type=pa.bool_()),
+        pa.array([{"a": 1, "b": "x"}, None, {"a": None, "b": None}],
+                 type=pa.struct([("a", pa.int32()), ("b", pa.string())])),
+        pa.array([], type=pa.list_(pa.int32())),
+        pa.array([None, None], type=pa.list_(pa.string())),
+    ]
+    for arr in arrays:
+        for view in (arr, arr.slice(1), arr.slice(0, 2), arr.slice(2)):
+            assert view.to_pylist() == [x.as_py() for x in view]
+
+    # Values inside numeric lists must stay Python ints/None, never floats
+    result = pa.array([[1, None, 3]], type=pa.list_(pa.int32())).to_pylist()
+    assert result == [[1, None, 3]]
+    assert [type(x) for x in result[0]] == [int, type(None), int]
+
+    # Duplicate struct field names raise like StructScalar.as_py does
+    dup = pa.StructArray.from_arrays(
+        [pa.array([1, 2]), pa.array(["a", "b"])], names=["x", "x"])
+    with pytest.raises(ValueError):

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4919614637

   Note on the two red CI jobs — both are unrelated to this change:
   
   - **AMD64 macOS 15-intel Python 3**: the only failure is `test_s3fs_limited_permissions_create_bucket`, an S3/minio fixture issue (the `mc admin` subprocess dies with SIGPIPE before the filesystem is exercised). The same job passes 7616 tests, including all array/scalar/conversion tests. The same flake also hit the previous commit's run.
   - **AMD64 Conda Python 3.11 Sphinx & Numpydoc**: two doctest failures in code this PR doesn't touch — `Table.join_asof` (the example's output chunking varies with task scheduling: `[[1,3],[2,3,3]]` vs `[[1,3,2,3,3]]`) and `default_memory_pool` (environment-dependent). The same job passed on the first commit of this PR with the same doctests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-5047298625

   @github-actions crossbow submit -g python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4920118310

   Reviewed with an AI code-review pass (multi-reviewer logical + dead-code analysis). No critical, major, or minor findings — the 13 raw comments generated all resolved to false positives on validation (verified claims around `StructArray::field()` child slicing, `CListArray::values()` unsliced children, GIL-safe cache publication, and per-type `type_id` uniqueness) or were below the auto-post threshold. LGTM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4997337758

   @pitrou Hi, can you take another look at this PR? We are looking forward to get this in and a new pyarrow release with it. Can you help on this? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3629218729


##########
python/pyarrow/tests/test_array.py:
##########
@@ -465,6 +465,56 @@ def test_array_getitem_numpy_scalars():
         assert arr[np.int32(idx)].as_py() == lst[idx]
 
 
+def test_to_pylist_bulk_paths():
+    # GH-50326: to_pylist converts through scalar-free _getitem_py
+    # specializations; the result must match the per-scalar conversion
+    # exactly.
+    arrays = [
+        pa.array([[1, None, 3], None, [], [4]], type=pa.list_(pa.int32())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.list_(pa.string())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.large_list(pa.large_string())),
+        pa.array([[1, None], None, [3, 4]], type=pa.list_(pa.int32(), 2)),
+        pa.array([[[1], [2, None]], None, [None, [3]]],
+                 type=pa.list_(pa.list_(pa.int32()))),
+        pa.array([[("k1", 1), ("k2", None)], None, []],
+                 type=pa.map_(pa.string(), pa.int32())),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.string()),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.large_string()),
+        pa.array([b"a\x00b", None, b"", b"\xff"], type=pa.binary()),
+        pa.array([b"a\x00b", None, b""], type=pa.large_binary()),
+        pa.array([[b"x", None, b"\x00y"], None, []],
+                 type=pa.list_(pa.binary())),
+        pa.array([1, None, -(2**62), 2**62], type=pa.int64()),
+        pa.array([0, None, 2**63 + 7], type=pa.uint64()),
+        pa.array([-128, 127, None], type=pa.int8()),
+        pa.array([1.5, None, -0.5], type=pa.float64()),
+        pa.array([1.5, None], type=pa.float32()),
+        pa.array([True, None, False], type=pa.bool_()),
+        pa.array([{"a": 1, "b": "x"}, None, {"a": None, "b": None}],
+                 type=pa.struct([("a", pa.int32()), ("b", pa.string())])),
+        pa.array([], type=pa.list_(pa.int32())),
+        pa.array([None, None], type=pa.list_(pa.string())),
+    ]
+    for arr in arrays:
+        for view in (arr, arr.slice(1), arr.slice(0, 2), arr.slice(2)):
+            assert view.to_pylist() == [x.as_py() for x in view]
+
+    # Values inside numeric lists must stay Python ints/None, never floats
+    result = pa.array([[1, None, 3]], type=pa.list_(pa.int32())).to_pylist()
+    assert result == [[1, None, 3]]
+    assert [type(x) for x in result[0]] == [int, type(None), int]
+
+    # Duplicate struct field names raise like StructScalar.as_py does
+    dup = pa.StructArray.from_arrays(
+        [pa.array([1, 2]), pa.array(["a", "b"])], names=["x", "x"])
+    with pytest.raises(ValueError):

Review Comment:
   I don't see this change, did you forget to push?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-5047582260

   @pitrou Heads-up: I pushed one more small commit (9b1863dc75) after your approval — the Copilot review on the stacked #50430 noticed that the `value_offset`/`value_length` declarations in libarrow.pxd take `int` while the C++ signatures take `int64_t`, so the new `_getitem_py` list/map paths would silently truncate row indices beyond INT_MAX. The fix just aligns the declarations with the C++ headers (no functional change below 2^31). Apologies for the extra CI round.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4861214624

   Good question — Cython already knows. Its type inference marks `result = []` as `list`, so the untyped version and an explicit `cdef list result = []` compile to the *identical* generated C: both lower `result.append(x)` to Cython's inlined `__Pyx_PyList_Append()`, which appends in place via `PyList_SET_ITEM` while there's spare capacity and only falls back to `PyList_Append` (geometric resize) otherwise. I verified by cythonizing both variants side by side and diffing the generated C — the loop bodies are byte-identical. That's indeed why the `append` form is already fast, and why adding `cdef list` doesn't move the numbers.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4861115156

   Okay if the benchmark is similar this is good. Would defining result as a list help? Like a cdef list for it and do append. Just curious whether cython knows it's a list already - maybe it does and that's why it's fast.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3552416254


##########
python/pyarrow/array.pxi:
##########
@@ -4229,6 +4357,30 @@ cdef class StructArray(Array):
     Concrete class for Arrow arrays of a struct data type.
     """
 
+    cdef object _getitem_py(self, int64_t i):
+        if self.ap.IsNull(i):
+            return None
+        cdef int64_t k, num_fields = self.type.num_fields
+        if self._children_cache is None:
+            names = [self.type.field(k).name for k in range(num_fields)]
+            if len(set(names)) != len(names):
+                # StructScalar.as_py raises ValueError on duplicate field
+                # names; mark the cache so we take the Scalar path below.
+                self._children_cache = (None, None)

Review Comment:
   Why not raise `ValueError` here instead of adding this weird fallback path?



##########
python/pyarrow/array.pxi:
##########
@@ -4006,6 +4105,15 @@ cdef class LargeStringArray(Array):
     Concrete class for Arrow arrays of large string (or utf8) data type.
     """
 
+    cdef object _getitem_py(self, int64_t i):
+        cdef:
+            int64_t length
+            const uint8_t* data
+        if self.ap.IsNull(i):
+            return None
+        data = (<CLargeStringArray*> self.ap).GetValue(i, &length)

Review Comment:
   You may be able to call `GetView(i)` which will give you a `std::string_view`.



##########
python/pyarrow/array.pxi:
##########
@@ -2458,6 +2476,34 @@ cdef class NumericArray(Array):
     A base class for Arrow numeric arrays.
     """
 
+    cdef object _getitem_py(self, int64_t i):
+        cdef Type tid = self.ap.type_id()
+        if self.ap.IsNull(i):
+            return None
+        if tid == _Type_INT64:
+            return (<CInt64Array*> self.ap).Value(i)
+        elif tid == _Type_INT32:
+            return (<CInt32Array*> self.ap).Value(i)

Review Comment:
   Why the peculiar ordering of types? I would rather have something more regular for readability.



##########
python/pyarrow/array.pxi:
##########
@@ -4006,6 +4105,15 @@ cdef class LargeStringArray(Array):
     Concrete class for Arrow arrays of large string (or utf8) data type.
     """
 
+    cdef object _getitem_py(self, int64_t i):
+        cdef:
+            int64_t length
+            const uint8_t* data
+        if self.ap.IsNull(i):
+            return None
+        data = (<CLargeStringArray*> self.ap).GetValue(i, &length)
+        return cp.PyUnicode_DecodeUTF8(<const char*> data, length, NULL)

Review Comment:
   We do support 32-bit platforms actually. But obviously a string size couldn't be larger than 4GiB on such a platform anyway, so the AI's comment is moot.



##########
python/pyarrow/tests/test_array.py:
##########
@@ -465,6 +465,56 @@ def test_array_getitem_numpy_scalars():
         assert arr[np.int32(idx)].as_py() == lst[idx]
 
 
+def test_to_pylist_bulk_paths():
+    # GH-50326: to_pylist converts through scalar-free _getitem_py
+    # specializations; the result must match the per-scalar conversion
+    # exactly.
+    arrays = [
+        pa.array([[1, None, 3], None, [], [4]], type=pa.list_(pa.int32())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.list_(pa.string())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.large_list(pa.large_string())),
+        pa.array([[1, None], None, [3, 4]], type=pa.list_(pa.int32(), 2)),
+        pa.array([[[1], [2, None]], None, [None, [3]]],
+                 type=pa.list_(pa.list_(pa.int32()))),
+        pa.array([[("k1", 1), ("k2", None)], None, []],
+                 type=pa.map_(pa.string(), pa.int32())),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.string()),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.large_string()),
+        pa.array([b"a\x00b", None, b"", b"\xff"], type=pa.binary()),
+        pa.array([b"a\x00b", None, b""], type=pa.large_binary()),
+        pa.array([[b"x", None, b"\x00y"], None, []],
+                 type=pa.list_(pa.binary())),
+        pa.array([1, None, -(2**62), 2**62], type=pa.int64()),
+        pa.array([0, None, 2**63 + 7], type=pa.uint64()),
+        pa.array([-128, 127, None], type=pa.int8()),
+        pa.array([1.5, None, -0.5], type=pa.float64()),
+        pa.array([1.5, None], type=pa.float32()),
+        pa.array([True, None, False], type=pa.bool_()),
+        pa.array([{"a": 1, "b": "x"}, None, {"a": None, "b": None}],
+                 type=pa.struct([("a", pa.int32()), ("b", pa.string())])),
+        pa.array([], type=pa.list_(pa.int32())),
+        pa.array([None, None], type=pa.list_(pa.string())),
+    ]

Review Comment:
   Can we do `pa.binary_view` as well?



##########
python/pyarrow/tests/test_array.py:
##########
@@ -465,6 +465,56 @@ def test_array_getitem_numpy_scalars():
         assert arr[np.int32(idx)].as_py() == lst[idx]
 
 
+def test_to_pylist_bulk_paths():
+    # GH-50326: to_pylist converts through scalar-free _getitem_py
+    # specializations; the result must match the per-scalar conversion
+    # exactly.
+    arrays = [
+        pa.array([[1, None, 3], None, [], [4]], type=pa.list_(pa.int32())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.list_(pa.string())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.large_list(pa.large_string())),
+        pa.array([[1, None], None, [3, 4]], type=pa.list_(pa.int32(), 2)),
+        pa.array([[[1], [2, None]], None, [None, [3]]],
+                 type=pa.list_(pa.list_(pa.int32()))),
+        pa.array([[("k1", 1), ("k2", None)], None, []],
+                 type=pa.map_(pa.string(), pa.int32())),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.string()),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.large_string()),
+        pa.array([b"a\x00b", None, b"", b"\xff"], type=pa.binary()),
+        pa.array([b"a\x00b", None, b""], type=pa.large_binary()),
+        pa.array([[b"x", None, b"\x00y"], None, []],
+                 type=pa.list_(pa.binary())),
+        pa.array([1, None, -(2**62), 2**62], type=pa.int64()),
+        pa.array([0, None, 2**63 + 7], type=pa.uint64()),
+        pa.array([-128, 127, None], type=pa.int8()),
+        pa.array([1.5, None, -0.5], type=pa.float64()),
+        pa.array([1.5, None], type=pa.float32()),
+        pa.array([True, None, False], type=pa.bool_()),
+        pa.array([{"a": 1, "b": "x"}, None, {"a": None, "b": None}],
+                 type=pa.struct([("a", pa.int32()), ("b", pa.string())])),
+        pa.array([], type=pa.list_(pa.int32())),
+        pa.array([None, None], type=pa.list_(pa.string())),
+    ]
+    for arr in arrays:
+        for view in (arr, arr.slice(1), arr.slice(0, 2), arr.slice(2)):
+            assert view.to_pylist() == [x.as_py() for x in view]
+
+    # Values inside numeric lists must stay Python ints/None, never floats
+    result = pa.array([[1, None, 3]], type=pa.list_(pa.int32())).to_pylist()
+    assert result == [[1, None, 3]]
+    assert [type(x) for x in result[0]] == [int, type(None), int]
+
+    # Duplicate struct field names raise like StructScalar.as_py does
+    dup = pa.StructArray.from_arrays(
+        [pa.array([1, 2]), pa.array(["a", "b"])], names=["x", "x"])
+    with pytest.raises(ValueError):

Review Comment:
   Can you test (part of) the error message as well? Something like:
   
   ```suggestion
       with pytest.raises(ValueError, match='some regex'):
   ```



##########
python/pyarrow/array.pxi:
##########
@@ -1864,7 +1864,19 @@ cdef class Array(_PandasConvertible):
         lst : list
         """
         self._assert_cpu()
-        return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        cdef int64_t i, n = self.length()
+        if maps_as_pydicts is not None:
+            # Converting maps to dicts has per-entry semantics (duplicate-key
+            # detection); use the Scalar-based conversion for exact behavior.
+            return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        return [self._getitem_py(i) for i in range(n)]
+
+    cdef object _getitem_py(self, int64_t i):

Review Comment:
   Not necessary for this PR, but as the AI hinted it might be better to replace this with a `cdef list _getitem_range_py(self, int64_t offset, int64_t length)`. This would cut down on function call and prologue overhead.
   
   Perhaps add a TODO or open a separate issue?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3630948471


##########
python/pyarrow/tests/test_array.py:
##########
@@ -465,6 +465,56 @@ def test_array_getitem_numpy_scalars():
         assert arr[np.int32(idx)].as_py() == lst[idx]
 
 
+def test_to_pylist_bulk_paths():
+    # GH-50326: to_pylist converts through scalar-free _getitem_py
+    # specializations; the result must match the per-scalar conversion
+    # exactly.
+    arrays = [
+        pa.array([[1, None, 3], None, [], [4]], type=pa.list_(pa.int32())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.list_(pa.string())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.large_list(pa.large_string())),
+        pa.array([[1, None], None, [3, 4]], type=pa.list_(pa.int32(), 2)),
+        pa.array([[[1], [2, None]], None, [None, [3]]],
+                 type=pa.list_(pa.list_(pa.int32()))),
+        pa.array([[("k1", 1), ("k2", None)], None, []],
+                 type=pa.map_(pa.string(), pa.int32())),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.string()),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.large_string()),
+        pa.array([b"a\x00b", None, b"", b"\xff"], type=pa.binary()),
+        pa.array([b"a\x00b", None, b""], type=pa.large_binary()),
+        pa.array([[b"x", None, b"\x00y"], None, []],
+                 type=pa.list_(pa.binary())),
+        pa.array([1, None, -(2**62), 2**62], type=pa.int64()),
+        pa.array([0, None, 2**63 + 7], type=pa.uint64()),
+        pa.array([-128, 127, None], type=pa.int8()),
+        pa.array([1.5, None, -0.5], type=pa.float64()),
+        pa.array([1.5, None], type=pa.float32()),
+        pa.array([True, None, False], type=pa.bool_()),
+        pa.array([{"a": 1, "b": "x"}, None, {"a": None, "b": None}],
+                 type=pa.struct([("a", pa.int32()), ("b", pa.string())])),
+        pa.array([], type=pa.list_(pa.int32())),
+        pa.array([None, None], type=pa.list_(pa.string())),
+    ]

Review Comment:
   You're right, apologies — 3d303ce487's commit message described these test changes but `test_array.py` never made it into that commit, and I replied without re-checking the pushed diff. Now actually added in 1d80beb925 (string_view/binary_view cases covering inline and out-of-line values; verified the view specializations engage — ~13x faster than the per-Scalar path on 1M rows).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3630954278


##########
python/pyarrow/array.pxi:
##########
@@ -1864,7 +1864,21 @@ cdef class Array(_PandasConvertible):
         lst : list
         """
         self._assert_cpu()
-        return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        cdef int64_t i, n = self.length()
+        if maps_as_pydicts is not None:
+            # Converting maps to dicts has per-entry semantics (duplicate-key
+            # detection); use the Scalar-based conversion for exact behavior.
+            return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]

Review Comment:
   @rok Sounds good — filed GH-50604. For the record on current behavior: a top-level `to_pylist` override on a subclass still dispatches normally via Python method resolution; the bypass only concerns *nested* children, where the previous behavior was already inconsistent — list children went through `child.to_pylist()` (override honored) while struct/map children were converted via per-field Scalars (override ignored). `ExtensionScalar.as_py` / `__arrow_ext_scalar_class__` remains honored on all paths, before and after this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4915939607

   Thanks for the review, @pitrou! Agreed — the per-type `to_pylist` overrides don't generalize. I'll rework the PR along your suggestion, which I read as a scalar-free `getitem`:
   
   1. Add `cdef object _getitem_py(self, int64_t i, ...)` on `Array`, whose base implementation preserves today's semantics exactly (`GetScalar(i)` + `as_py()`), so any type without a specialization behaves identically to today.
   2. Make the baseline `Array.to_pylist` a single loop over `_getitem_py`.
   3. Specialize `_getitem_py` for the common types: integers, floats, boolean, string/binary (+ large variants), list-likes (build each row's list directly from the child's `_getitem_py` over the offset range), struct and map — nested types then compose without allocating any Scalar or Array wrapper per row, and `maps_as_pydicts` threads through as an argument.
   
   Types with non-trivial `as_py` semantics (dates/timestamps/decimals/dictionary/...) would keep the exact fallback initially and can be specialized incrementally in follow-ups; your `int64` example above lands in the specialized bucket and should get within striking distance of `ndarray.tolist`.
   
   One question before I rework: do you prefer this mechanism in Cython as sketched, or as a C++ `ToPyList` visitor under `python/pyarrow/src/arrow/python/` (in the spirit of `MonthDayNanoIntervalArrayToPyList`)? The Cython version keeps the unspecialized fallback trivially exact and the diff reviewable; the C++ visitor is likely faster still (and would also serve `ChunkedArray`/`Table` uniformly) but is a much bigger surface. Happy to go either way.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3629216807


##########
python/pyarrow/tests/test_array.py:
##########
@@ -465,6 +465,56 @@ def test_array_getitem_numpy_scalars():
         assert arr[np.int32(idx)].as_py() == lst[idx]
 
 
+def test_to_pylist_bulk_paths():
+    # GH-50326: to_pylist converts through scalar-free _getitem_py
+    # specializations; the result must match the per-scalar conversion
+    # exactly.
+    arrays = [
+        pa.array([[1, None, 3], None, [], [4]], type=pa.list_(pa.int32())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.list_(pa.string())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.large_list(pa.large_string())),
+        pa.array([[1, None], None, [3, 4]], type=pa.list_(pa.int32(), 2)),
+        pa.array([[[1], [2, None]], None, [None, [3]]],
+                 type=pa.list_(pa.list_(pa.int32()))),
+        pa.array([[("k1", 1), ("k2", None)], None, []],
+                 type=pa.map_(pa.string(), pa.int32())),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.string()),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.large_string()),
+        pa.array([b"a\x00b", None, b"", b"\xff"], type=pa.binary()),
+        pa.array([b"a\x00b", None, b""], type=pa.large_binary()),
+        pa.array([[b"x", None, b"\x00y"], None, []],
+                 type=pa.list_(pa.binary())),
+        pa.array([1, None, -(2**62), 2**62], type=pa.int64()),
+        pa.array([0, None, 2**63 + 7], type=pa.uint64()),
+        pa.array([-128, 127, None], type=pa.int8()),
+        pa.array([1.5, None, -0.5], type=pa.float64()),
+        pa.array([1.5, None], type=pa.float32()),
+        pa.array([True, None, False], type=pa.bool_()),
+        pa.array([{"a": 1, "b": "x"}, None, {"a": None, "b": None}],
+                 type=pa.struct([("a", pa.int32()), ("b", pa.string())])),
+        pa.array([], type=pa.list_(pa.int32())),
+        pa.array([None, None], type=pa.list_(pa.string())),
+    ]

Review Comment:
   I don't see the added cases, did you forget to push?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3630168835


##########
python/pyarrow/array.pxi:
##########
@@ -1864,7 +1864,21 @@ cdef class Array(_PandasConvertible):
         lst : list
         """
         self._assert_cpu()
-        return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        cdef int64_t i, n = self.length()
+        if maps_as_pydicts is not None:
+            # Converting maps to dicts has per-entry semantics (duplicate-key
+            # detection); use the Scalar-based conversion for exact behavior.
+            return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]

Review Comment:
   Let's declare this out of scope and capture it with a new issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-5049164133

   Thank you @pitrou @rok


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3629728508


##########
python/pyarrow/array.pxi:
##########
@@ -1864,7 +1864,21 @@ cdef class Array(_PandasConvertible):
         lst : list
         """
         self._assert_cpu()
-        return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        cdef int64_t i, n = self.length()
+        if maps_as_pydicts is not None:
+            # Converting maps to dicts has per-entry semantics (duplicate-key
+            # detection); use the Scalar-based conversion for exact behavior.
+            return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]

Review Comment:
   Won't this proposal also ignore user-defined `CustomArray(pa.ExtensionArray).to_pylist` here?
   How about we check for custom `to_pylist` and use this path too. See sketch below:
   
   ```diff
   diff --git i/python/pyarrow/array.pxi w/python/pyarrow/array.pxi
   index 71c18fbdfe..523a603b29 100644
   --- i/python/pyarrow/array.pxi
   +++ w/python/pyarrow/array.pxi
   @@ -1123,6 +1123,17 @@ cdef PandasOptions _convert_pandas_options(dict options):
        return result
   
   
   +cdef bint _has_custom_extension_to_pylist(DataType type):
   +    cdef int i
   +    if type.id == _Type_EXTENSION:
   +        array_class = type.__arrow_ext_class__()
   +        return array_class.to_pylist is not ExtensionArray.to_pylist
   +    for i in range(type.num_fields):
   +        if _has_custom_extension_to_pylist(type.field(i).type):
   +            return True
   +    return False
   +
   +
    cdef class Array(_PandasConvertible):
        """
        The base class for all Arrow arrays.
   @@ -1865,9 +1876,11 @@ cdef class Array(_PandasConvertible):
            """
            self._assert_cpu()
            cdef int64_t i, n = self.length()
   -        if maps_as_pydicts is not None:
   +        if (maps_as_pydicts is not None or
   +                _has_custom_extension_to_pylist(self.type)):
                # Converting maps to dicts has per-entry semantics (duplicate-key
   -            # detection); use the Scalar-based conversion for exact behavior.
   +            # detection), while extension arrays can customize to_pylist;
   +            # use the Scalar-based conversion for exact behavior.
                return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
            # TODO(GH-50448): convert per range instead of per element to cut
            # the per-element call overhead further.
   diff --git i/python/pyarrow/tests/test_extension_type.py w/python/pyarrow/tests/test_extension_type.py
   index 35b801eca8..3f5eca5222 100644
   --- i/python/pyarrow/tests/test_extension_type.py
   +++ w/python/pyarrow/tests/test_extension_type.py
   @@ -391,6 +391,21 @@ def test_ext_array_to_pylist():
        assert arr.to_pylist() == [b"foo", b"bar", None]
   
   
   +def test_nested_custom_ext_array_to_pylist():
   +    class CustomArray(pa.ExtensionArray):
   +        def to_pylist(self, **kwargs):
   +            return ["custom"] * len(self)
   +
   +    class CustomType(IntegerType):
   +        def __arrow_ext_class__(self):
   +            return CustomArray
   +
   +    values = pa.ExtensionArray.from_storage(CustomType(), pa.array([1]))
   +    arr = pa.ListArray.from_arrays([0, 1], values)
   +
   +    assert arr.to_pylist() == [["custom"]]
   +
   +
    def test_ext_array_errors():
        ty = ParamExtType(4)
        storage = pa.array([b"foo", b"bar"], type=pa.binary(3))
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3629254266


##########
python/pyarrow/array.pxi:
##########
@@ -1864,7 +1864,21 @@ cdef class Array(_PandasConvertible):
         lst : list
         """
         self._assert_cpu()
-        return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        cdef int64_t i, n = self.length()
+        if maps_as_pydicts is not None:
+            # Converting maps to dicts has per-entry semantics (duplicate-key
+            # detection); use the Scalar-based conversion for exact behavior.
+            return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]

Review Comment:
   It's a pity that we're taking the slow path even if no maps are involved. Perhaps add a comment or TODO about this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4917636970

   Reworked as suggested — the PR now adds a scalar-free `cdef _getitem_py(self, int64_t i)` on `Array` and the baseline `to_pylist` is a single loop over it. The base implementation is `GetScalar + Scalar.as_py`, so unspecialized types (dates/times/timestamps/decimals/dictionary/extension/...) behave exactly as before, and specializations exist for integers, floats, boolean, string/binary (+ large variants), list/large_list/fixed_size_list, map and struct. Nested types compose: a list row is built directly from the child's `_getitem_py` over its offset range, with the child wrapper cached on the parent array. `maps_as_pydicts != None` keeps the Scalar path since map→dict conversion has per-entry duplicate-key semantics.
   
   Your `int64` example: 4M `int64` with nulls goes from ~0.39 s to **0.028 s (~7 ns/element — on par with `ndarray.tolist`)**. Other numbers (M4 Max): flat string 4M 0.83→0.06 s; `list<string>` 2M 1.93→0.46 s; `list<list<int32>>` 1M 2.10→0.40 s; `struct<int64,string>` 1M 0.91→0.07 s; `map<string,int64>` 1M 2.77→0.74 s.
   
   Net diff is smaller than the previous approach (−104 lines) since the per-type `to_pylist` overrides are gone. Differential tests against the Scalar path (exact type equality, incl. slices, all-null, duplicate struct field names raising `ValueError`, strict-mode maps) pass, as does the pytest suite (1295 passed). PR description updated accordingly.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3546240817


##########
python/pyarrow/array.pxi:
##########
@@ -4006,6 +4105,15 @@ cdef class LargeStringArray(Array):
     Concrete class for Arrow arrays of large string (or utf8) data type.
     """
 
+    cdef object _getitem_py(self, int64_t i):
+        cdef:
+            int64_t length
+            const uint8_t* data
+        if self.ap.IsNull(i):
+            return None
+        data = (<CLargeStringArray*> self.ap).GetValue(i, &length)
+        return cp.PyUnicode_DecodeUTF8(<const char*> data, length, NULL)

Review Comment:
   `PyUnicode_DecodeUTF8` expects a `Py_ssize_t` length; passing the `int64_t` length from `LargeStringArray::GetValue` can overflow/truncate on 32-bit builds or for very large values, potentially turning the length negative and causing crashes or incorrect decoding. Please cast via `Py_ssize_t` with an explicit overflow check (and apply the same pattern to `LargeBinaryArray._getitem_py`, which has the same issue).



##########
python/pyarrow/tests/test_array.py:
##########
@@ -465,6 +465,38 @@ def test_array_getitem_numpy_scalars():
         assert arr[np.int32(idx)].as_py() == lst[idx]
 
 
+def test_to_pylist_bulk_paths():
+    # GH-50326: list-like and string arrays convert to Python objects in
+    # bulk instead of going through one Scalar per element; the result must
+    # match the per-scalar conversion exactly.
+    arrays = [
+        pa.array([[1, None, 3], None, [], [4]], type=pa.list_(pa.int32())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.list_(pa.string())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.large_list(pa.large_string())),
+        pa.array([[1, None], None, [3, 4]], type=pa.list_(pa.int32(), 2)),
+        pa.array([[[1], [2, None]], None, [None, [3]]],
+                 type=pa.list_(pa.list_(pa.int32()))),
+        pa.array([[("k1", 1), ("k2", None)], None, []],
+                 type=pa.map_(pa.string(), pa.int32())),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.string()),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.large_string()),
+        pa.array([], type=pa.list_(pa.int32())),
+        pa.array([None, None], type=pa.list_(pa.string())),
+    ]

Review Comment:
   `BinaryArray`/`LargeBinaryArray` now have scalar-free `_getitem_py` fast paths, but this test only exercises string and list/string cases. Adding binary and list<binary> cases here would help ensure the bytes fast path matches scalar conversion (including embedded NUL bytes) and stays covered across slices.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4926061083

   @dbtsai For the record, are these comments AI-generated? Did you vet them for correctness?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-5047616643

   > @pitrou Heads-up: I pushed one more small commit ([9b1863d](https://github.com/apache/arrow/commit/9b1863dc750507d40de412f447bc8275651cdba7)) after your approval — the Copilot review on the stacked #50430 noticed that the `value_offset`/`value_length` declarations in libarrow.pxd take `int` while the C++ signatures take `int64_t`, so the new `_getitem_py` list/map paths would silently truncate row indices beyond INT_MAX. The fix just aligns the declarations with the C++ headers (no functional change below 2^31).
   
   Thank you! I don't think it will change anything for CI, so I'll merge accordingly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3629784220


##########
python/pyarrow/array.pxi:
##########
@@ -1864,7 +1864,21 @@ cdef class Array(_PandasConvertible):
         lst : list
         """
         self._assert_cpu()
-        return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        cdef int64_t i, n = self.length()
+        if maps_as_pydicts is not None:
+            # Converting maps to dicts has per-entry semantics (duplicate-key
+            # detection); use the Scalar-based conversion for exact behavior.
+            return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]

Review Comment:
   Is overriding `to_pylist` even supported?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-5068268351

   After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 0083d11f61446ee57c260ce1af5e2a39ac745d48.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/89446819102) has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3546201025


##########
python/pyarrow/lib.pxd:
##########
@@ -282,6 +282,8 @@ cdef class Array(_PandasConvertible):
     cdef:
         shared_ptr[CArray] sp_array
         CArray* ap
+        # Lazily wrapped child array(s) reused by _getitem_py (see GH-50326)
+        object _children_cache

Review Comment:
   Adding `_children_cache` inside the first `cdef:` block changes the memory layout of `Array` *before* existing fields like `type`/`_name`. That can break ABI for downstream Cython extensions that were compiled against an older `pyarrow/lib.pxd` but run against this version (their compiled field offsets no longer match). To reduce ABI risk, keep existing field offsets stable by moving `_children_cache` to the end of the class layout (e.g., in a second `cdef:` block after the `cdef readonly:` fields).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3546430562


##########
python/pyarrow/lib.pxd:
##########
@@ -282,6 +282,8 @@ cdef class Array(_PandasConvertible):
     cdef:
         shared_ptr[CArray] sp_array
         CArray* ap
+        # Lazily wrapped child array(s) reused by _getitem_py (see GH-50326)
+        object _children_cache

Review Comment:
   Good point — moved the declaration after the pre-existing attributes in 728584f8e6, so `type`/`_name` offsets stay stable and the new field follows the append-only convention that Cython's size check assumes.



##########
python/pyarrow/array.pxi:
##########
@@ -4006,6 +4105,15 @@ cdef class LargeStringArray(Array):
     Concrete class for Arrow arrays of large string (or utf8) data type.
     """
 
+    cdef object _getitem_py(self, int64_t i):
+        cdef:
+            int64_t length
+            const uint8_t* data
+        if self.ap.IsNull(i):
+            return None
+        data = (<CLargeStringArray*> self.ap).GetValue(i, &length)
+        return cp.PyUnicode_DecodeUTF8(<const char*> data, length, NULL)

Review Comment:
   PyArrow only supports 64-bit platforms, where `Py_ssize_t` and `int64_t` have the same width; and on a hypothetical 32-bit build a single value longer than `PY_SSIZE_T_MAX` could not exist in the first place, since the child data buffer itself is bounded by the process address space. So the implicit conversion cannot truncate in practice.



##########
python/pyarrow/tests/test_array.py:
##########
@@ -465,6 +465,38 @@ def test_array_getitem_numpy_scalars():
         assert arr[np.int32(idx)].as_py() == lst[idx]
 
 
+def test_to_pylist_bulk_paths():
+    # GH-50326: list-like and string arrays convert to Python objects in
+    # bulk instead of going through one Scalar per element; the result must
+    # match the per-scalar conversion exactly.
+    arrays = [
+        pa.array([[1, None, 3], None, [], [4]], type=pa.list_(pa.int32())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.list_(pa.string())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.large_list(pa.large_string())),
+        pa.array([[1, None], None, [3, 4]], type=pa.list_(pa.int32(), 2)),
+        pa.array([[[1], [2, None]], None, [None, [3]]],
+                 type=pa.list_(pa.list_(pa.int32()))),
+        pa.array([[("k1", 1), ("k2", None)], None, []],
+                 type=pa.map_(pa.string(), pa.int32())),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.string()),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.large_string()),
+        pa.array([], type=pa.list_(pa.int32())),
+        pa.array([None, None], type=pa.list_(pa.string())),
+    ]

Review Comment:
   Added binary/large_binary (including embedded NUL bytes), list<binary>, wide-range integers, floats, boolean and struct coverage in 728584f8e6, plus an assertion that duplicate struct field names still raise `ValueError`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3509800101


##########
python/pyarrow/array.pxi:
##########
@@ -3974,6 +4155,45 @@ cdef class StringArray(Array):
     Concrete class for Arrow arrays of string (or utf8) data type.
     """
 
+    def to_pylist(self, *, maps_as_pydicts=None):
+        """
+        Convert to a list of native Python objects.
+
+        Parameters
+        ----------
+        maps_as_pydicts : str, optional, default `None`
+            Valid values are `None`, 'lossy', or 'strict'.
+            This parameter is ignored for non-nested Arrays.
+
+        Returns
+        -------
+        lst : list
+        """
+        cdef:
+            CStringArray* arr = <CStringArray*> self.ap
+            int64_t i, n
+            int32_t length
+            const uint8_t* data
+        self._assert_cpu()
+        n = arr.length()
+        result = []
+        # Decode values straight from the data buffer instead of creating
+        # a C++ Scalar and a Python Scalar wrapper per value (see GH-28694).
+        if arr.null_count() == 0:
+            for i in range(n):
+                data = arr.GetValue(i, &length)
+                result.append(
+                    cp.PyUnicode_DecodeUTF8(<const char*> data, length, NULL))
+        else:
+            for i in range(n):
+                if arr.IsNull(i):
+                    result.append(None)
+                else:
+                    data = arr.GetValue(i, &length)
+                    result.append(
+                        cp.PyUnicode_DecodeUTF8(<const char*> data, length, NULL))
+        return result

Review Comment:
   `null_count()` is a one-time vectorized popcount over the validity bitmap (~n/8 bytes, well under a millisecond for 2M rows), computed and cached per `ArrayData`. In exchange, the no-null branch skips the per-element `IsNull()` check entirely. Branching on `null_bitmap_data() == NULL` instead would save that single scan but degrade the common case of a sliced/combined array that *has* a bitmap yet contains no nulls in range — that would take the per-element `IsNull()` path forever. So the current form should be the better trade-off in practice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4912480927

   > The cause is the per-element conversion loop (`[x.as_py() for x in self]`): every row allocates a C++ Scalar (`Array::GetScalar`), a Python Scalar wrapper and, for list types, a Python Array wrapper for the row's values slice plus a fresh generator before recursing per element.
   
   I agree this is extremely wasteful, but the approach here seems very ad hoc and also solves the performance issue for a very limited set of types, while the performance problem is more general:
   ```python
   In [5]: a = np.arange(10_0000)
   
   In [6]: %timeit a.tolist()
   722 μs ± 43.8 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
   
   In [7]: b = pa.array(a)
   
   In [8]: %timeit b.to_pylist()
   17.5 ms ± 147 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
   
   In [9]: b.to_pylist() == a.tolist()
   Out[9]: True
   ```
   
   How about we do something like the following:
   1) Add an `Array` method that indexes and returns a Python value (e.g. an `int`) directly instead of a PyArrow scalar (e.g. an Int64Scalar)
   2) Use that new method in the baseline implementation of `Array::to_pylist`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3629841938


##########
python/pyarrow/array.pxi:
##########
@@ -1864,7 +1864,21 @@ cdef class Array(_PandasConvertible):
         lst : list
         """
         self._assert_cpu()
-        return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        cdef int64_t i, n = self.length()
+        if maps_as_pydicts is not None:
+            # Converting maps to dicts has per-entry semantics (duplicate-key
+            # detection); use the Scalar-based conversion for exact behavior.
+            return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]

Review Comment:
   I doubt this was discussed before.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-5047383658

   Revision: 1d80beb92573fa60163c6f88bd5b435c92970ad1
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-08c8d9798b](https://github.com/ursacomputing/crossbow/branches/all?query=actions-08c8d9798b)
   
   |Task|Status|
   |----|------|
   |example-python-minimal-build-fedora-conda|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-example-python-minimal-build-fedora-conda)](https://github.com/ursacomputing/crossbow/actions/runs/29928249993/job/88950876444)|
   |example-python-minimal-build-ubuntu-venv|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-example-python-minimal-build-ubuntu-venv)](https://github.com/ursacomputing/crossbow/actions/runs/29928255296/job/88950893135)|
   |test-conda-python-3.10|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.10)](https://github.com/ursacomputing/crossbow/actions/runs/29928251692/job/88950881968)|
   |test-conda-python-3.10-hdfs-2.9.2|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.10-hdfs-2.9.2)](https://github.com/ursacomputing/crossbow/actions/runs/29928255184/job/88950892309)|
   |test-conda-python-3.10-hdfs-3.2.1|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.10-hdfs-3.2.1)](https://github.com/ursacomputing/crossbow/actions/runs/29928252053/job/88950882736)|
   |test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2)](https://github.com/ursacomputing/crossbow/actions/runs/29928249392/job/88950874551)|
   |test-conda-python-3.11|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.11)](https://github.com/ursacomputing/crossbow/actions/runs/29928255284/job/88950892410)|
   |test-conda-python-3.11-dask-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.11-dask-latest)](https://github.com/ursacomputing/crossbow/actions/runs/29928252025/job/88950882584)|
   |test-conda-python-3.11-dask-upstream_devel|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.11-dask-upstream_devel)](https://github.com/ursacomputing/crossbow/actions/runs/29928251946/job/88950882546)|
   |test-conda-python-3.11-hypothesis|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.11-hypothesis)](https://github.com/ursacomputing/crossbow/actions/runs/29928250842/job/88950879007)|
   |test-conda-python-3.11-pandas-latest-numpy-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.11-pandas-latest-numpy-latest)](https://github.com/ursacomputing/crossbow/actions/runs/29928251656/job/88950881643)|
   |test-conda-python-3.11-spark-master|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.11-spark-master)](https://github.com/ursacomputing/crossbow/actions/runs/29928249924/job/88950876286)|
   |test-conda-python-3.12|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.12)](https://github.com/ursacomputing/crossbow/actions/runs/29928250169/job/88950876721)|
   |test-conda-python-3.12-pandas-latest-numpy-1.26|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.12-pandas-latest-numpy-1.26)](https://github.com/ursacomputing/crossbow/actions/runs/29928252984/job/88950886001)|
   |test-conda-python-3.12-pandas-latest-numpy-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.12-pandas-latest-numpy-latest)](https://github.com/ursacomputing/crossbow/actions/runs/29928248707/job/88950872798)|
   |test-conda-python-3.13|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.13)](https://github.com/ursacomputing/crossbow/actions/runs/29928255292/job/88950893180)|
   |test-conda-python-3.13-pandas-nightly-numpy-nightly|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.13-pandas-nightly-numpy-nightly)](https://github.com/ursacomputing/crossbow/actions/runs/29928247482/job/88950869093)|
   |test-conda-python-3.13-pandas-upstream_devel-numpy-nightly|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.13-pandas-upstream_devel-numpy-nightly)](https://github.com/ursacomputing/crossbow/actions/runs/29928248098/job/88950870572)|
   |test-conda-python-3.14|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.14)](https://github.com/ursacomputing/crossbow/actions/runs/29928250065/job/88950876287)|
   |test-conda-python-3.14-cpython-debug|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-3.14-cpython-debug)](https://github.com/ursacomputing/crossbow/actions/runs/29928248620/job/88950872398)|
   |test-conda-python-emscripten|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-conda-python-emscripten)](https://github.com/ursacomputing/crossbow/actions/runs/29928251498/job/88950881073)|
   |test-debian-13-python-3-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-debian-13-python-3-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/29928255210/job/88950892535)|
   |test-debian-13-python-3-i386|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-debian-13-python-3-i386)](https://github.com/ursacomputing/crossbow/actions/runs/29928255922/job/88950894456)|
   |test-fedora-42-python-3|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-fedora-42-python-3)](https://github.com/ursacomputing/crossbow/actions/runs/29928252633/job/88950884627)|
   |test-ubuntu-22.04-python-3|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-ubuntu-22.04-python-3)](https://github.com/ursacomputing/crossbow/actions/runs/29928247018/job/88950867136)|
   |test-ubuntu-24.04-python-3|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-08c8d9798b-github-test-ubuntu-24.04-python-3)](https://github.com/ursacomputing/crossbow/actions/runs/29928254331/job/88950889613)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3630950218


##########
python/pyarrow/tests/test_array.py:
##########
@@ -465,6 +465,56 @@ def test_array_getitem_numpy_scalars():
         assert arr[np.int32(idx)].as_py() == lst[idx]
 
 
+def test_to_pylist_bulk_paths():
+    # GH-50326: to_pylist converts through scalar-free _getitem_py
+    # specializations; the result must match the per-scalar conversion
+    # exactly.
+    arrays = [
+        pa.array([[1, None, 3], None, [], [4]], type=pa.list_(pa.int32())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.list_(pa.string())),
+        pa.array([["a", None], None, [], ["bcd", ""]],
+                 type=pa.large_list(pa.large_string())),
+        pa.array([[1, None], None, [3, 4]], type=pa.list_(pa.int32(), 2)),
+        pa.array([[[1], [2, None]], None, [None, [3]]],
+                 type=pa.list_(pa.list_(pa.int32()))),
+        pa.array([[("k1", 1), ("k2", None)], None, []],
+                 type=pa.map_(pa.string(), pa.int32())),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.string()),
+        pa.array(["a", None, "", "\N{GRINNING FACE} \N{SNOWMAN}"],
+                 type=pa.large_string()),
+        pa.array([b"a\x00b", None, b"", b"\xff"], type=pa.binary()),
+        pa.array([b"a\x00b", None, b""], type=pa.large_binary()),
+        pa.array([[b"x", None, b"\x00y"], None, []],
+                 type=pa.list_(pa.binary())),
+        pa.array([1, None, -(2**62), 2**62], type=pa.int64()),
+        pa.array([0, None, 2**63 + 7], type=pa.uint64()),
+        pa.array([-128, 127, None], type=pa.int8()),
+        pa.array([1.5, None, -0.5], type=pa.float64()),
+        pa.array([1.5, None], type=pa.float32()),
+        pa.array([True, None, False], type=pa.bool_()),
+        pa.array([{"a": 1, "b": "x"}, None, {"a": None, "b": None}],
+                 type=pa.struct([("a", pa.int32()), ("b", pa.string())])),
+        pa.array([], type=pa.list_(pa.int32())),
+        pa.array([None, None], type=pa.list_(pa.string())),
+    ]
+    for arr in arrays:
+        for view in (arr, arr.slice(1), arr.slice(0, 2), arr.slice(2)):
+            assert view.to_pylist() == [x.as_py() for x in view]
+
+    # Values inside numeric lists must stay Python ints/None, never floats
+    result = pa.array([[1, None, 3]], type=pa.list_(pa.int32())).to_pylist()
+    assert result == [[1, None, 3]]
+    assert [type(x) for x in result[0]] == [int, type(None), int]
+
+    # Duplicate struct field names raise like StructScalar.as_py does
+    dup = pa.StructArray.from_arrays(
+        [pa.array([1, 2]), pa.array(["a", "b"])], names=["x", "x"])
+    with pytest.raises(ValueError):

Review Comment:
   Same mistake as the sibling thread, apologies — added in 1d80beb925: `pytest.raises(ValueError, match="duplicate field names")`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3630952037


##########
python/pyarrow/array.pxi:
##########
@@ -1864,7 +1864,21 @@ cdef class Array(_PandasConvertible):
         lst : list
         """
         self._assert_cpu()
-        return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]
+        cdef int64_t i, n = self.length()
+        if maps_as_pydicts is not None:
+            # Converting maps to dicts has per-entry semantics (duplicate-key
+            # detection); use the Scalar-based conversion for exact behavior.
+            return [x.as_py(maps_as_pydicts=maps_as_pydicts) for x in self]

Review Comment:
   Added a TODO in 1d80beb925. Note the stacked #50430 removes this limitation entirely — it threads `maps_as_pydicts` through `_getitem_py` so every type keeps its fast path when the flag is set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-5047956815

   @pitrou One more declaration-alignment commit (95f29b9b5b), prompted by further Copilot findings on the stacked #50430: my earlier fix only covered `value_offset`/`value_length`, but the same stale-`int` issue applied to `IsNull`/`Value`/`GetValue`/`GetString`/`FormatValue` — `IsNull` in particular is called with `int64_t` indices by every `_getitem_py`. All element accessors now match the C++ headers; I swept the whole file this time, so this should be the last of the family. Sorry for the churn.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #50327:
URL: https://github.com/apache/arrow/pull/50327


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4916577681

   > One question before I rework: do you prefer this mechanism in Cython as sketched, or as a C++ `ToPyList` visitor under `python/pyarrow/src/arrow/python/` (in the spirit of `MonthDayNanoIntervalArrayToPyList`)? The Cython version keeps the unspecialized fallback trivially exact and the diff reviewable; the C++ visitor is likely faster still (and would also serve `ChunkedArray`/`Table` uniformly) but is a much bigger surface. Happy to go either way.
   
   Doing this in Cython would probably be more maintainable, right? Wrangling CPython API calls from C++ is generally tedious and error-prone.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4920289691

   **Follow-up ideas: further performance opportunities**
   
   The structural win here (no per-element Scalar/Array allocation + child-array caching) is solid. A few places that still leave performance on the table, roughly by impact:
   
   1. **Hoist the numeric type-ID dispatch out of the per-element loop (biggest).** `NumericArray._getitem_py` recomputes `type_id()` and walks an `if/elif` ladder (up to 10 branches) on *every* element, even though the type is invariant across the array. For an int8 column that's ~6 failed comparisons per value. Resolving the type once and running a monomorphic loop over the buffer turns the "no Scalar" win into a "vectorized" win.
   
   2. **Share the fast path with `as_py()` / `__iter__`.** `_getitem_py` is private and only called from `to_pylist` (when `maps_as_pydicts is None`). `arr[i].as_py()`, iteration, and `ChunkedArray` element access still allocate Scalars. Wiring those to reuse `_getitem_py` would extend the win to the other common access patterns.
   
   3. **Null fast path.** Every accessor calls `IsNull(i)` per element even when `null_count == 0`. Checking `null_count == 0` once and skipping the per-element null test helps the very common all-non-null column.
   
   4. **Offset reads in list/map paths.** `value_offset(i)` / `value_offset(i+1)` are two virtual calls per row; reading the offsets buffer pointer once and indexing avoids that on long arrays. Also worth confirming `for j in range(start, end)` compiles to a C loop rather than allocating a `range` per row.
   
   5. **Missing specializations that still fall back to Scalars:** temporal (date/timestamp/time/duration), decimal, `DictionaryArray`, and string_view/binary_view have no `_getitem_py`. Timestamps/dates and dictionaries are common enough to be worth a fast path (dictionary especially — see the separate value-reuse comment).
   
   None of these are blocking — the PR is a clear improvement as-is. #1 and #2 are the two with real leverage.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4920283598

   **Follow-up idea: reuse decoded strings for dictionary-encoded arrays**
   
   The new `StringArray._getitem_py` calls `PyUnicode_DecodeUTF8` per element, so a repeated string value produces a fresh `str` object every time (N allocations + N UTF-8 decodes for N elements). Two places where value reuse could help:
   
   1. **`DictionaryArray` (highest value, no correctness risk):** the dictionary already stores each distinct value once. A specialized path could decode the D dictionary entries once and then return interned references (`decoded[index[i]]` + `Py_INCREF`) — turning N decodes into D decodes + N pointer reuses. This PR currently doesn't specialize `DictionaryArray`, so it falls back to the Scalar path and this win is unclaimed. Since `str` is immutable, sharing references is semantically identical to copying.
   
   2. **Plain `StringArray` with repeated values:** an explicit dedup cache (or `PyUnicode_InternInPlace`) would cut allocations on low-cardinality data, but adds a hash/lookup per element and regresses high-cardinality (mostly-unique) data. Probably only worth it behind a cardinality heuristic / opt-in, not by default.
   
   Not blocking for this PR — just flagging `DictionaryArray` as a natural, safe follow-up where value reuse is a large win.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50327:
URL: https://github.com/apache/arrow/pull/50327#discussion_r3509608761


##########
python/pyarrow/array.pxi:
##########
@@ -3974,6 +4155,45 @@ cdef class StringArray(Array):
     Concrete class for Arrow arrays of string (or utf8) data type.
     """
 
+    def to_pylist(self, *, maps_as_pydicts=None):
+        """
+        Convert to a list of native Python objects.
+
+        Parameters
+        ----------
+        maps_as_pydicts : str, optional, default `None`
+            Valid values are `None`, 'lossy', or 'strict'.
+            This parameter is ignored for non-nested Arrays.
+
+        Returns
+        -------
+        lst : list
+        """
+        cdef:
+            CStringArray* arr = <CStringArray*> self.ap
+            int64_t i, n
+            int32_t length
+            const uint8_t* data
+        self._assert_cpu()
+        n = arr.length()
+        result = []
+        # Decode values straight from the data buffer instead of creating
+        # a C++ Scalar and a Python Scalar wrapper per value (see GH-28694).
+        if arr.null_count() == 0:
+            for i in range(n):
+                data = arr.GetValue(i, &length)
+                result.append(
+                    cp.PyUnicode_DecodeUTF8(<const char*> data, length, NULL))
+        else:
+            for i in range(n):
+                if arr.IsNull(i):
+                    result.append(None)
+                else:
+                    data = arr.GetValue(i, &length)
+                    result.append(
+                        cp.PyUnicode_DecodeUTF8(<const char*> data, length, NULL))
+        return result

Review Comment:
   Using `arr.null_count() == 0` can force an extra full validity-bitmap scan when `null_count` is unknown (e.g., slices of arrays with any nulls), because `null_count()` computes and caches the count. Since the else-branch already calls `IsNull(i)` per element, this can add an avoidable extra pass; consider branching on `arr.null_bitmap_data() == NULL` (definitely no nulls) and otherwise doing per-element `IsNull(i)` without calling `null_count()` (apply similarly to LargeStringArray / list bulk paths).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4860856583

   :warning: GitHub issue #50326 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4860959014

   I'm not an expert in Cython but curious about how `result.append()` works. I think as long as `result` is not defined as a Cython `list`, `result.append` still works as a pure Python object? In that case, each `append` would trigger a dynamic attribute search and the list would be reallocated a few times during appending.
   
   It would be an interesting experiment to do a full-allocation for the list before assigning the data, as we already know the length of the list. An extra step forward is to declare the return value as a list in Cython so it can optimize `setitem` even more.
   
   Something like
   
   ```cython
   cdef list result = [None] * n
   cdef Py_ssize_t i
   
   for i in range(n):
       result[i] = ...
   
   return result
   ```
   
   For a long list this might push the performance even further.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Speed up to_pylist for list-like and string arrays [arrow]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4861011814

   Good idea — I tried exactly that (`cdef list result = [None] * n` + indexed assignment, also letting null rows keep the prefilled `None`), but it benchmarks the same or slightly worse: `list<string>` 0.34 s → 0.35–0.37 s, `list<list<int32>>` 0.65 s → 0.69–0.70 s (best of 3, consistent across reruns).
   
   Two reasons, I think: Cython already lowers `result.append(x)` on a local it can see is a list to a `PyList_Append` call, and CPython lists over-allocate geometrically, so the resize cost is amortized and tiny. Meanwhile the preallocated version pays an extra refcount round-trip per slot (every `result[i] = x` has to decref the prefilled `None`). The loops are dominated by creating the per-row slice objects / decoding UTF-8 either way, so I kept the simpler `append` form.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist [arrow]

Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #50327:
URL: https://github.com/apache/arrow/pull/50327#issuecomment-4928442510

   > @dbtsai For the record, are these comments AI-generated? Did you vet them for correctness?
   
   Those were AI-generated based on my prompts for specific areas where I had questions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org